Intelligent POI search method based on multi-modal intention analysis and knowledge graph

Abstract

The invention discloses an intelligent POI search method based on multi-modal intention analysis and a knowledge graph. The method comprises the steps that multi-modal query information input by a user is received and verified; carrying out preprocessing and reconstruction query on the multi-modal data; an encoder fusion strategy is adopted to form unified multi-modal feature representation, and a deep network based on an attention mechanism is utilized to perform intention recognition and analysis; interacting with the knowledge graph through a vectorization retrieval mode, and carrying out semantic expansion and confirmation on the intention; according to the complexity of the intention, dynamic task scheduling and distributed parallel retrieval are carried out in combination with the real-time state of the system; and finally, carrying out personalized result reordering by using a learning ordering model and fusing multi-dimensional features. Through the method of combining multi-modal fusion, knowledge graph interaction and resource self-adaptive scheduling, the problems of one-sided intention understanding, shallow knowledge utilization and low search efficiency in the prior art are solved, and accurate, efficient and personalized POI search is realized.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, information retrieval, and geographic information systems, and in particular to an intelligent point of interest (POI) search method based on multimodal intent analysis and knowledge graphs. Background Technology

[0002] With the rapid development of mobile internet and location services, point-of-interest (POI) search has become a core entry point for digital life. For users with diverse input styles and varied query intentions, efficient and intelligent search methods will greatly facilitate their search experience.

[0003] Traditional POI search technologies primarily rely on keyword matching and simple geographic filtering. Their core flaw lies in treating user queries as isolated strings rather than semantic carriers of complex intent. They struggle to understand users' complex, implicit query intentions, performing poorly, especially in multimodal input scenarios. The limitations of current mainstream search methods are reflected in the following three dimensions: Firstly, at the intent understanding level, most only process text queries, or while supporting multimodal input, they fail to perform deep integration and accurate intent parsing. They fragment key intent signals from the original modality, such as emotion, style, and scene composition. For example, if a user takes a photo of a "cafe with outdoor seating" and says "looking for this kind of atmosphere," existing systems often lose unstructured visual and emotional features like "outdoor" and "atmosphere," causing search results to deviate from the essential needs. Secondly, at the knowledge utilization level, they rely on static attribute databases or only use knowledge graphs for interpretation after retrieval, failing to perform dynamic semantic association and reasoning during the query understanding stage, making it difficult to meet complex needs such as "suitable for bringing children." Finally, in terms of architectural efficiency, the fixed processing flow cannot distinguish between simple and complex queries, resulting in suboptimal allocation of system resources and insufficient response latency or processing depth in high-concurrency scenarios.

[0004] In summary, there is an urgent need for an intelligent POI search solution that can deeply integrate multimodal information, accurately interpret user intent, and efficiently execute personalized searches. Summary of the Invention

[0005] This invention aims to overcome the shortcomings of existing technologies and provide an intelligent POI search method based on multimodal intent analysis and knowledge graphs. By constructing a complete technical chain from multimodal adaptive preprocessing, deep fusion parsing, graph vectorization interaction to resource-aware scheduling, it integrates core components such as multimodal information verification and preprocessing, unified feature representation fusion, intent recognition and parsing based on deep networks, semantic retrieval and expansion of vectorized knowledge graphs, dynamic task decomposition and parallel scheduling, and personalized learning, ranking, and re-ranking. This achieves accurate parsing of users' complex intents and dynamic mining of implicit needs. Ultimately, it achieves the goal of intelligently optimizing the search process and resource utilization efficiency, thereby comprehensively improving the accuracy, intelligence level, and overall system performance of POI search services.

[0006] The technical solution adopted by this invention to achieve its objective is as follows:

[0007] A smart POI search method based on multimodal intent analysis and knowledge graph includes the following steps:

[0008] S1: Receives raw query data in multimodal form (text, voice, or image) input by the user and performs format and security checks;

[0009] S2: For the verified multimodal data, we preprocess it using an adaptive word segmentation model, an end-to-end speech recognition model, and a visual understanding model, and then reconstruct the query using syntactic analysis and query expansion techniques to obtain the reconstructed multimodal data.

[0010] S3: Encode the preprocessed multimodal data into feature vectors, and use a feature layer fusion method of concatenation and projection (referring to dimensionality reduction and nonlinear transformation) to form a unified multimodal joint feature representation;

[0011] S4: Input the multimodal joint feature representation and real-time context information into a deep network based on the Transformer decoder for intent recognition and parsing, and output a structured intent description with confidence.

[0012] S5: Map structured intent descriptions to query vectors, perform semantic retrieval and expansion in the vectorized knowledge graph through approximate nearest neighbor search, and output enhanced structured intent objects;

[0013] S6: Analyze the complexity of the enhanced structured intent object, dynamically decompose it into a directed acyclic graph of subtasks, combine the performance status of distributed nodes to perform task scheduling and parallel index retrieval, and aggregate the results;

[0014] S7: Employs a learning-based ranking model that integrates query matching degree, user personalization features, and contextual features to re-rank the aggregated results and generate the final search results list.

[0015] Furthermore, in step S2, the preprocessing methods for different types of input data are as follows: the adaptive word segmentation model uses the BERT model's word segmenter; the end-to-end speech recognition model is the OpenAI Whisper model; and the visual understanding model is a target detection and scene classification model based on Vision Transformer.

[0016] Furthermore, the query reconstruction in step S2, involving syntactic analysis and query expansion, involves: performing syntactic structure analysis on the preprocessed data, including word segmentation, stop word removal, phrase identification, and part-of-speech tagging, converting it into a form that the search engine can understand, and extracting key information; and optimizing the query statement through synonym replacement, spelling correction, and query expansion to improve the relevance and accuracy of the search.

[0017] Furthermore, the feature layer fusion method in step S3 specifically involves concatenating the text feature vector, the speech emotion feature vector, and the visual semantic feature vector, and then performing dimensionality reduction and nonlinear transformation through a multilayer perceptron to generate a unified multimodal joint feature representation.

[0018] Furthermore, the deep network based on the Transformer decoder described in step S4 generates a structured intent sequence through self-attention and cross-attention mechanisms, and introduces user history behavior, geographical location, time and weather as contextual features for intent refinement and personalized inference.

[0019] Furthermore, in step S5, the vectorized knowledge graph is created by using the TransE knowledge graph embedding model to map entities and relations into vectors; semantic retrieval and expansion are achieved by calculating the similarity between the intent query vector and the entity and relation vectors in the graph, and then using approximate nearest neighbor search to quickly find relevant nodes, thereby associating and supplementing implicit constraints.

[0020] Furthermore, the scheduling criteria for task scheduling in step S6 include the number of constraints in the enhanced structured intent object, whether multi-hop graph reasoning is required, and the real-time CPU and memory load indicators of each distributed search node; parallel index retrieval is executed based on horizontally sharded inverted indexes.

[0021] Furthermore, in step S7, the ranking model learned is the LambdaMART model, whose ranking feature set includes: the matching score between the candidate object and the structured intent, user historical preference features, real-time context relevance features, and objective quality features.

[0022] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps of the method as claimed in any one of claims 1 to 8.

[0023] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements an intelligent POI search method based on multimodal intent analysis and knowledge graph.

[0024] Compared with the prior art, the advantages of the present invention are as follows:

[0025] (1) Significantly improved depth and accuracy of intent understanding: By introducing multiple pre-trained models from the preprocessing stage and using attention-based deep networks in the fusion and parsing stages, accurate capture and alignment of fine-grained semantics across modalities are achieved, which can effectively understand abstract and complex intents such as “atmosphere” and “suitable scene”.

[0026] (2) Knowledge reasoning achieves dynamism and intelligence: The innovative “vectorized retrieval” method is used to interact with the knowledge graph, transforming the static knowledge base into a dynamic resource that can efficiently perform semantic similarity calculation, realizing rapid and accurate semantic expansion of user intent, and enabling the search system to have certain “cognition” and “association” capabilities.

[0027] (3) Optimization of system resource efficiency and scalability: By using fine-grained task splitting based on query complexity and dynamic scheduling strategy with resource awareness, combined with distributed index with horizontal sharding, the system can intelligently adapt to different loads, ensuring millisecond-level response for massive simple queries and ensuring that complex queries are processed in depth through parallelization, thus achieving the best balance between throughput, latency and search depth.

[0028] (4) User experience closed loop optimization: The final ranking process deeply integrates the precise intent, long-term user preferences and real-time scenario information parsed from the original query, ensuring that the search results are not only relevant, but also highly personalized and contextualized, forming a complete user experience closed loop of "precise understanding - intelligent retrieval - optimized presentation". Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the main process of an intelligent POI search method based on multimodal intent analysis and knowledge graph according to the present invention. Detailed Implementation

[0030] The following is in conjunction with the appendix Figure 1 The present invention will be further described in conjunction with specific embodiments.

[0031] This invention discloses an intelligent POI search method based on multimodal intent analysis and knowledge graph. First, it receives multimodal retrieval information from the user, such as text, voice, or images. It then preprocesses the input data for different types, performing preliminary analysis and rewriting of the query information. Next, it fuses and extracts features from the preprocessed multimodal data. Based on the extracted features, it identifies and parses the intent, interacts with and expands the knowledge graph, and outputs structured intent information. According to the intent and system status, it assigns the query to appropriate search nodes. Finally, it aggregates the results, collecting results from multiple search nodes, merging, deduplicating, sorting, and personalizing them to generate search results that better meet the user's needs.

[0032] The specific steps are as follows:

[0033] S1: Receives and validates multimodal raw query data. Receives user-submitted multimodal raw query data (text, voice, images, or combinations thereof) via the system interface, and performs format validation and basic security scanning to ensure valid input.

[0034] S2: Preprocessing and query reconstruction of validated multimodal data. A parallel, specialized workflow is employed to process each modality: text is segmented and entity recognized using a BERT pre-trained model; speech is transcribed using a Whisper end-to-end model for high-precision transcription and acoustic sentiment analysis; and images are detected and understood using a Vision Transformer model. Furthermore, the preprocessed textual content (text, speech, and images) undergoes syntactic analysis, spell correction, and query expansion based on thesaurus or language models to complete query reconstruction.

[0035] S3: Feature Representation Learning Based on Encoder Fusion. The preprocessed multimodal data is encoded into feature vectors. The preprocessed text feature vectors, speech emotion vectors, and visual semantic vectors are concatenated. Then, a multilayer perceptron is used for dimensionality reduction and nonlinear transformation to generate a unified multimodal joint feature representation. In this process, the model can adaptively learn the weights of each modality.

[0036] S4: Deep Intent Parsing Based on Attention Mechanism. The multimodal joint feature representations described above, along with features extracted from user profiles and real-time context (location, time, etc.), are input into a sequence generation model based on a Transformer decoder deep network. This model uses an attention mechanism to identify and parse intent, outputting a structured intent description sequence (e.g., "Action: Search, Subject: Restaurant, Constraints: {Average Price: 200-300 RMB, Atmosphere: Quiet}"), and assigning a confidence score to each element.

[0037] S5: Knowledge Graph Interaction and Intent Structuring Based on Vectorized Retrieval. The TransE knowledge graph embedding model is used to encode structured intents into query vectors. Within the vectorized domain knowledge graph, for each query vector, a near-nearest neighbor search is used to quickly find semantically similar entities and relational nodes, thereby associating and expanding the implicit needs of the query vector (e.g., associating "family dinner" with "providing child seats"). The final output is an enhanced structured intent object (e.g., in JSON-LD format).

[0038] S6: Complexity-Based Dynamic Task Scheduling and Parallel Retrieval. The dynamic task scheduler analyzes the complexity of the enhanced structured intent object (number of constraints, inference depth, etc.) and decomposes it into a directed acyclic graph of subtasks. Combined with real-time performance monitoring of the distributed search node cluster, subtasks are dynamically allocated to the nodes with the optimal load. Each node accesses its locally maintained horizontally sharded inverted index in parallel, performs retrieval, returns preliminary results, and aggregates the retrieval results from all nodes.

[0039] S7: Personalized Learning Ranking Based on Multi-Feature Fusion. The LambdaMART ranking model is used for final ranking. This model integrates query matching features, user personalization features, contextual features, and business rule features to calculate the final score for each candidate result, generating and returning a highly personalized list of search results.

[0040] Example 1: Processing complex multimodal restaurant search requests.

[0041] Suppose the user input is: a picture of a restaurant private room with a large round table and karaoke equipment, and the voice message "Looking for a place that can seat about 15 people and is suitable for company team building".

[0042] S1-S2 (Reception and Preprocessing): The system receives image and audio files, verifies them, and processes them in parallel. Images are processed using the Vision Transformer model to extract visual semantic tags such as [large round table, karaoke equipment, private room, Chinese-style decoration]. Audio is transcribed into text using the Whisper model, and the acoustic analysis module adds sentiment tags such as "formal" and "lively." Dependency parsing is performed on the transcribed text to identify core components, and the term "team building" is expanded to "team building, gathering" using a thesaurus.

[0043] S3-S4 (Feature Fusion and Intent Resolution): Text features, visual label features, and voice emotion features are concatenated and fused to generate a joint feature representation. This feature, along with the user profile (business entertainment consumption records) and the current context (Friday evening 19:00), is input into the Transformer decoder. The decoder outputs a structured intent:

[0044] {Action: Search, Subject: Restaurant, Constraints: {Capacity: ≈15, Table Type: Large Round Table, Facilities: [Private Rooms, Karaoke]}, Scenario: Company Team Building, Implicit Needs: [High Privacy, Multimedia Equipment], Confidence: 0.92}

[0045] S5 (Knowledge Graph Interaction): The above intent is mapped into a query vector using the TransE encoder. An approximate nearest neighbor search is performed in the vectorized catering knowledge graph to find the "team building" node, and further associate it with nodes such as "requires projector," "requires professional sound system," and "high tolerance for ambient noise." The output is an enhanced intent object, supplemented with the specific requirement of "multimedia equipment."

[0046] S6 (Dynamic Scheduling and Retrieval): The scheduler determines that the intent object is highly complex (multiple constraints, requiring association with implicit attributes). It breaks it down into four sub-tasks: geographic filtering, matching of private rooms and karaoke facilities, capacity verification, and brand preference filtering. These sub-tasks are dynamically distributed to three search nodes for parallel execution based on real-time node load. Each node queries its local horizontal shard index and returns a set of candidate restaurants.

[0047] S7 (Personalized Ranking): After aggregating the candidate set, the LambdaMART ranking model works. It calculates the characteristics of each restaurant: matching score with "large round table" and "karaoke", user's historical preference score for "Hunan cuisine", cost score based on the user's current distance, and reservation popularity score for Friday evening, etc. The model integrates these characteristics and ranks those restaurants that fully meet the explicit conditions, partially meet the implicit conditions (such as having a projector), and match the user's tastes at the top, generating the final list.

[0048] Example 2: Handling Simple and Convenient Inquiries

[0049] The user enters plain text: "Nearby 24-hour pharmacies".

[0050] S1-S4: The system quickly verifies and processes text. Through BERT word segmentation and encoding, combined with the user's real-time geolocation, the intent parsing model rapidly outputs structured intent: {Action: Search, Subject: Pharmacy, Key Constraint: Open 24 hours, Ranking Preference: Nearest}.

[0051] S5: After the intent is mapped to a vector, the relationship between the "pharmacy" entity and the "business hours" attribute is quickly identified in the knowledge graph, and a simple enhanced intent object is generated.

[0052] S6-S7: The scheduler recognizes this as a low-complexity query and, to reduce scheduling overhead, directly assigns it to a high-concurrency, low-latency fast retrieval node. This node uses a spatial index to perform proximity retrieval and filters for business hours attributes, returning the results directly sorted by distance for maximum response speed.

Claims

1. A multi-modal intent analysis and knowledge graph based intelligent POI search method, characterized in that, The method comprises the following steps: S1: receiving user input text, voice or image multi-modal original query data, and performing format and security verification; S2: for the verified multi-modal data, respectively using an adaptive word segmentation model, an end-to-end speech recognition model and a visual understanding model for preprocessing, and through syntax analysis and query expansion technology for query reconstruction, obtaining multi-modal data after query reconstruction; S3: encoding the preprocessed multi-modal data into a feature vector, using a feature layer fusion method after splicing and projection to form a unified multi-modal joint feature representation; S4: inputting the multi-modal joint feature representation and real-time context information into a deep network based on a Transformer decoder for intent recognition and analysis, and outputting a structured intent description with confidence; S5: mapping the structured intent description to a query vector, and performing semantic retrieval and expansion in the vectorized knowledge graph through approximate nearest neighbor search, and outputting an enhanced structured intent object; S6: analyzing the complexity of the enhanced structured intent object, dynamically splitting it into a directed acyclic graph of subtasks, combining the distributed node performance state for task scheduling and parallel index retrieval, and aggregating the results; S7: using a learning ranking model to re-rank the aggregated results by combining query matching degree, user personalized features and context features, and generating a final search result list.

2. The intelligent POI search method of claim 1, wherein, In step S2, the corresponding preprocessing methods for different types of input data are as follows: the adaptive word segmentation model uses the tokenizer of the BERT model; the end-to-end speech recognition model is the OpenAI Whisper model; and the visual understanding model is a target detection and scene classification model based on Vision Transformer. 3.The intelligent POI search method of claim 1, wherein, In step S2, the query reconstruction of syntax analysis and query expansion is: performing syntax structure analysis on the preprocessed data, including word segmentation, removing stop words, identifying phrases, part-of-speech tagging, converting into a form understood by the search engine, and performing key information extraction; through synonym replacement, spelling correction and query expansion, the query sentence is optimized to improve the relevance and accuracy of the search.

4. The intelligent POI search method of claim 1, wherein, In step S3, the feature layer fusion method is to splice the text feature vector, speech emotion feature vector and visual semantic feature vector, and perform dimension reduction and nonlinear transformation through a multilayer perceptron to generate a unified multi-modal joint feature representation. 5.The intelligent POI search method of claim 1, wherein, In step S4, the deep network based on the Transformer decoder generates a structured intent sequence through self-attention and cross-attention mechanisms, and introduces user historical behavior, geographic location and time weather as context features for intent refinement and personalized inference. 6.The intelligent POI search method of claim 1, wherein, In step S5, the vectorized knowledge graph is a TransE knowledge graph embedding model that maps entities and relationships into vectors; semantic retrieval and expansion is to calculate the similarity of the intent query vector and the entity and relationship vectors in the graph, and use approximate nearest neighbor search to quickly find related nodes, thereby associating and supplementing implicit constraints. 7.The intelligent POI search method of claim 1, wherein, The scheduling of the task scheduling in step S6 is based on the number of constraints in the enhanced structured intent object, whether multi-hop graph reasoning is needed, and real-time CPU and memory load indicators of each distributed search node; and parallel index retrieval is performed based on horizontally sharded inverted indexes. 8.The intelligent POI search method of claim 1, wherein, The learning ranking model in step S7 is a LambdaMART model, and the ranking feature set thereof includes: a matching score of the candidate object and the structured intent, user historical preference features, real-time context relevance features, and objective quality features.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 8 when executing the program.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the steps of the method of any one of claims 1 to 8 when executed by the processor.

Images