Scene recommendation method based on multi-modal context driving and related equipment
By constructing an initial candidate set, a spatiotemporal transition candidate set, and a preference candidate set through a multimodal context-driven scene recommendation method, and using a large language model to predict target scene interest points, this method solves the problems of rigid fusion strategies and insufficient logical reasoning in existing interest point recommendation methods, and achieves personalized recommendations with high accuracy and low workload.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SOUTH CHINA NORMAL UNIV
- Filing Date
- 2025-12-19
- Publication Date
- 2026-05-01
AI Technical Summary
Existing interest point recommendation methods suffer from rigid, hard-coded information fusion strategies and a lack of logical reasoning ability, resulting in recommendation results that do not match the scenario and have low accuracy. Furthermore, they rely on large-scale data fine-tuning, which leads to a large workload and makes it difficult to adapt to changes in user interests.
By employing a multimodal context-driven scenario recommendation method, we construct an initial candidate set, a spatiotemporal transition candidate set, and a preference candidate set. We then utilize a large language model to predict target scenario interest points, dynamically adjust information priorities, reduce model fine-tuning, and improve recommendation accuracy.
It improves the accuracy of recommendation results and reduces workload without updating model parameters, provides clear explanations of recommendations, and adapts to the dynamic needs of users.
Smart Images
Figure CN121958671A_ABST
Abstract
Description
Multimodal context-driven scene recommendation methods and related devices Technical Field
[0001] This application relates to the field of intelligent recommendation technology, and in particular to a scene recommendation method and related device based on multimodal context-driven approach. Background Technology
[0002] In related technologies, Points of Interest (POI) recommendation aims to accurately predict a user's next likely destination, thereby providing each user with highly personalized and contextualized travel and lifestyle decision support. Existing POI recommendation methods, due to their hard-coded and static fusion strategies, result in an overly rigid information fusion process in any scenario, potentially leading to recommendation results that do not match the current context. Furthermore, existing recommendation processes heavily rely on model fine-tuning on large-scale data, resulting in a significant workload. In addition, existing recommendation methods predict based on learning statistical correlations in data, lacking logical reasoning capabilities, thus failing to understand the user's true intent during the recommendation process, leading to low accuracy.
[0003] In summary, the technical problems existing in the relevant technologies need to be improved. Summary of the Invention
[0004] The main objective of this application is to propose a scene recommendation method and related device based on multimodal context-driven approach, which can effectively improve the accuracy of recommendation results and reduce the workload of the recommendation process.
[0005] To achieve the above objectives, one aspect of this application proposes a scene recommendation method based on multimodal context-driven approach. The method includes the following steps: acquiring a multimodal database of points of interest (POIs) and the current user's current recommendation request, current trajectory data, and historical trajectory data; constructing an initial candidate set based on the current recommendation request and the POI database, wherein all POIs in the initial candidate set are related to the current user's current interests; performing spatiotemporal transition retrieval on the initial candidate set based on the historical trajectory data to obtain a spatiotemporal transition candidate set; performing preference retrieval on the initial candidate set based on the current trajectory data to obtain the current user's preference feature information and preference candidate set; constructing a prompt word template based on the current trajectory data, the historical trajectory data, the preference feature information, the preference candidate set, and the spatiotemporal transition candidate set; and inputting the prompt word template into a large language model to predict the current user's target scene POIs to obtain a target scene recommendation result.
[0006] In some embodiments, constructing an initial candidate set based on the current recommendation request and the multimodal interest database includes: jointly encoding the image information and label text information of each interest point in the multimodal interest database using a preset cross-modal alignment model to obtain image and text vector representations; storing the image and text vector representations corresponding to all interest points in a vector database and creating a FAISS index for the vector database; when the current recommendation request is received, generating a query vector based on the current user's current interests; and performing a similarity search in the vector database based on the query vector and the FAISS index to obtain several initial interest points that form the initial candidate set.
[0007] In some embodiments, the step of performing spatiotemporal transition retrieval on the initial candidate set based on the historical trajectory data to obtain a spatiotemporal transition candidate set includes: constructing a heterogeneous spatiotemporal graph based on the historical trajectory data, wherein the nodes of the heterogeneous spatiotemporal graph are the initial points of interest in the initial candidate set, and the edges of the heterogeneous spatiotemporal graph are provided with type information based on the relationship between the nodes; performing collaborative knowledge mining on the heterogeneous spatiotemporal graph to obtain fused spatiotemporal knowledge; and filtering the initial points of interest in the initial candidate set based on the fused spatiotemporal knowledge to obtain the moment points of interest that conform to the current user dynamic behavior trend, which constitute the spatiotemporal transition candidate set.
[0008] In some embodiments, the step of collaboratively mining knowledge from the heterogeneous spatiotemporal graph to obtain fused spatiotemporal knowledge includes: analyzing the transitional edges in the heterogeneous spatiotemporal graph to obtain knowledge of the transitional patterns of the current user between different initial points of interest; analyzing the spatial proximity edges in the heterogeneous spatiotemporal graph to obtain spatial knowledge corresponding to the geographical activity range and regional preferences of the current user; analyzing the temporal rhythm edges in the heterogeneous spatiotemporal graph to obtain temporal knowledge corresponding to the behavioral cycle pattern of the current user; and performing cross-dimensional fusion and reasoning on the transitional pattern knowledge, the spatial knowledge, and the temporal knowledge to obtain structured fused spatiotemporal knowledge.
[0009] In some embodiments, the step of performing preference retrieval on the initial candidate set based on the current trajectory data to obtain the current user's preference feature information and preference candidate set includes: obtaining comment information and interaction record information corresponding to each initial point of interest in the initial candidate set under the current trajectory data; analyzing the comment information and interaction record information to obtain the current user's preference feature information; performing semantic analysis on the comment information to obtain attribute feature information, the attribute feature information including semantic summary information and sentiment attribute information; evaluating the fit between the preference feature information and the attribute feature information corresponding to each initial candidate point in the initial candidate set; and filtering the initial candidate points in the initial candidate set according to the evaluated fit to obtain the preference candidate set.
[0010] In some embodiments, the step of inputting the prompt word template into a large language model to predict the target scene interest points of the current user and obtain a target scene recommendation result includes: inputting the prompt word template into the large language model, dynamically evaluating the relative importance of multi-source information in the prompt word template through the context learning ability of the large language model; reordering the initial interest points in the initial candidate set according to the evaluated relative importance to obtain the target scene recommended interest points in the target scene recommendation result; and generating a recommendation explanation corresponding to the target scene recommended interest points in the target scene recommendation result through natural language.
[0011] In some embodiments, the method further includes the following steps: obtaining comparison data between the target scene recommendation results and the target scene actual access data; and updating the prompt word template based on the comparison data.
[0012] To achieve the above objectives, another aspect of this application proposes a scene recommendation device based on multimodal context-driven methods. The device includes: a first module for acquiring a multimodal database of points of interest and the current user's current recommendation request, current trajectory data, and historical trajectory data; a second module for constructing an initial candidate set based on the current recommendation request and the multimodal database of points of interest, wherein all points of interest in the initial candidate set are related to the current user's current interests; a third module for performing spatiotemporal transition retrieval on the initial candidate set based on the historical trajectory data to obtain a spatiotemporal transition candidate set; a fourth module for performing preference retrieval on the initial candidate set based on the current trajectory data to obtain the current user's preference feature information and preference candidate set; a fifth module for constructing a prompt word template based on the current trajectory data, the historical trajectory data, the preference feature information, the preference candidate set, and the spatiotemporal transition candidate set; and a sixth module for inputting the prompt word template into a large language model to predict the current user's target scene points of interest, thereby obtaining a target scene recommendation result.
[0013] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described above.
[0014] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods described above.
[0015] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer program product, including a computer program that, when executed by a processor, implements the aforementioned method.
[0016] The embodiments of this application include at least the following beneficial effects: This application provides a scene recommendation method and related device based on multimodal context-driven approach. This scheme constructs an initial candidate set based on the current recommendation request and a multimodal database of points of interest. Then, it performs spatiotemporal transition retrieval on the initial candidate set based on historical trajectory data to obtain a spatiotemporal transition candidate set. Finally, it performs preference retrieval on the initial candidate set based on current trajectory data to obtain the current user's preference feature information and preference candidate set. Then, it constructs a prompt word template based on the current trajectory data, historical trajectory data, preference feature information, preference candidate set, and spatiotemporal transition candidate set, ensuring that the information in the prompt word template corresponds to the current user's current scene. Next, it inputs the prompt word template into a large language model to predict the current user's target scene points of interest, thereby obtaining the target scene recommendation result. This leverages the context learning capability of the large language model to improve the accuracy of the prediction result, thus improving the accuracy of the recommendation result and reducing the workload of the recommendation process. Attached Figure Description
[0017] Figure 1 is a flowchart of the scene recommendation method based on multimodal context-driven provided in an embodiment of this application; Figure 2 is a complete implementation flowchart of the scene recommendation method based on multimodal context-driven provided in an embodiment of this application; Figure 3 is a schematic diagram of the construction of the initial candidate set provided in an embodiment of this application; Figure 4 is a flowchart of the generation of the spatiotemporal transition candidate set provided in an embodiment of this application; Figure 5 is a flowchart of the generation of the preference candidate set provided in an embodiment of this application; Figure 6 is a structural schematic diagram of the scene recommendation device based on multimodal context-driven provided in an embodiment of this application; Figure 7 is a hardware structure schematic diagram of the electronic device provided in an embodiment of this application. Detailed Implementation
[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.
[0019] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”
[0020] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.
[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.
[0022] Before providing a detailed description of the embodiments of this application, some of the nouns and terms used in the embodiments of this application will be explained first. The nouns and terms used in the embodiments of this application are subject to the following interpretations: Large Language Model (LLM) is a natural language processing technology based on deep learning. It is trained on massive amounts of text data and is capable of generating, understanding, and processing natural language text. The core characteristics of LLM include large-scale parameters, complex network structure, and powerful generalization ability. It is widely used in tasks such as text generation, translation, and question answering.
[0023] Faiss Index is a core component of a similarity search and clustering library developed by Facebook AI Research for efficiently processing large-scale vector data. Instead of a traditional database, Faiss encapsulates vector datasets as indexes.1 An index is a data structure that stores vectors, supporting fast retrieval and querying.1 When searching, you input a query vector, and the index returns the k most similar vectors.
[0024] Among related technologies, Points of Interest (POI) recommendation is a core and key technology supporting various online-offline integrated services. It aims to accurately predict a user's next likely destination, thereby providing each user with highly personalized and contextualized travel and lifestyle decision support. POI recommendation employs a series of machine learning methods, utilizing massive amounts of user movement trajectories to dynamically predict their travel intentions. It is currently widely used in intelligent systems such as map navigation, online tourism, and local lifestyle services.
[0025] Current POI recommendation technology mainly develops along three relatively independent but overlapping paths: 1) Recall technology based on multimodal content: To address the inherent sparsity of user behavior data, modern recommender systems widely incorporate multimodal information such as images, text, and tags of POIs. Among them, the vector retrieval architecture based on the dual-tower model is the mainstream solution in the industry. This architecture maps users and POIs to the same semantic space through independent encoders and uses Approximate Nearest Neighbor (ANN) search to achieve efficient candidate set recall. In recent years, powerful cross-modal alignment models, such as CLIP and its subsequent evolution SigLIP2, have achieved deep semantic alignment of image and text features through pre-training on massive image-text pairs, greatly improving the quality of multimodal retrieval. 2) Sequence modeling based on spatiotemporal trajectories: User movement behavior has significant spatiotemporal dependencies. To capture these complex dynamic patterns, the research field has evolved from early recurrent neural networks (RNNs) to advanced paradigms dominated by spatio-temporal graph neural networks (STGNNs). STGNNs learn representations rich in spatiotemporal context by constructing heterogeneous graphs to uniformly model users, POIs, regions, and their geographic, temporal, and sequential relationships. However, the performance of STGNNs is highly dependent on the graph's topology and performs poorly in sparse data scenarios. 3) Preference inference based on comment text: User comments are a valuable data source for mining their fine-grained preferences and sentiment tendencies.
[0026] As discussed above, existing Point of Interest (POI) recommendation methods primarily focus on improving the accuracy of predicting a user's next destination, while generally neglecting to deeply analyze and understand the numerous potential factors influencing user decisions. Specifically, the shortcomings of existing technologies are mainly reflected in three aspects: First, the information fusion mechanism is rigid: existing hybrid models typically employ a hard-coded, static fusion strategy. Whether through simple feature concatenation or complex attention networks, the weights and interaction methods of different information sources (such as spatiotemporal, multimodal, and commentary) are fixed after model training. This rigid mechanism cannot dynamically adjust the priority of different information based on the user's current specific context (e.g., whether commuting daily or traveling), making it difficult for recommendation results to adapt to complex and ever-changing real-world needs.
[0027] Second, there is a high dependence on model fine-tuning: To achieve optimal performance on specific recommendation tasks, existing models almost invariably require expensive fine-tuning on large-scale labeled data. This process is not only computationally costly but also makes it difficult for the model to quickly adapt to real-time changes in user interests. Once a user's preferences shift, the system must wait for the model to be retrained before it can respond, which is a major drawback in modern applications that strive for real-time personalization.
[0028] Third, a lack of deep semantic reasoning and interpretability: Traditional deep learning models are essentially powerful pattern matchers that make predictions by learning statistical correlations in data, but lack true logical reasoning capabilities. For example, they struggle to understand complex semantics in reviews like "This restaurant is a bit expensive, but the atmosphere is quiet," which carries a sense of trade-offs. Furthermore, these models typically operate in a "black box" mode, unable to provide clear and logical explanations for their recommendations, thus reducing user trust.
[0029] Therefore, it can be seen that existing interest point recommendation methods either involve a large amount of data processing or have low accuracy in the recommendation results.
[0030] In view of this, this application provides a scene recommendation method and related device based on multimodal context driving, which can effectively improve the accuracy of recommendation results and reduce the workload of the recommendation process.
[0031] The scene recommendation method based on multimodal context-driven architecture provided in this application relates to the field of intelligent recommendation technology. This method can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or in-vehicle terminal, but is not limited to these. The server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network. The software can be an application implementing the scene recommendation method based on multimodal context-driven architecture, but is not limited to the above forms.
[0032] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.
[0033] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.
[0034] The embodiments of this application are described in detail below with reference to the accompanying drawings: Figure 1 is an optional flowchart of a scene recommendation method based on multimodal context-driven approach provided by an embodiment of this application. The method in Figure 1 may include, but is not limited to, steps S110 to S160: Step S110: Obtain an interest point multimodal database and the current user's current recommendation request, current trajectory data, and historical trajectory data; Step S120: Construct an initial candidate set based on the current recommendation request and the interest point multimodal database, wherein the interest points in the initial candidate set are all related to the current user's current interests; Step S130: Perform spatiotemporal transition retrieval on the initial candidate set based on historical trajectory data to obtain a spatiotemporal transition candidate set; Step S140: Perform preference retrieval on the initial candidate set based on the current trajectory data to obtain the current user's preference feature information and preference candidate set; Step S150: Construct a prompt word template based on the current trajectory data, historical trajectory data, preference feature information, preference candidate set, and spatiotemporal transition candidate set; Step S160: Input the prompt word template into a large language model to predict the current user's target scene interest points to obtain the target scene recommendation result.
[0035] It is understood that the data in the multimodal interest database of this embodiment can be online global interest point data, including but not limited to data such as the geographical location or facilities, tags, and images of interest points. After obtaining the multimodal interest database, as shown in Figure 2, this embodiment constructs an initial candidate set based on the current recommendation request and the multimodal interest database. Specifically, as shown in Figure 3, this embodiment can use a preset cross-modal alignment model to jointly encode the image information and tag text information of each interest point in the multimodal interest database to obtain image and text vector representations. Then, the image and text vector representations corresponding to all interest points are stored in a vector database, and a FAISS index is created for the vector database. When the current recommendation request is received, a query vector is generated based on the current user's current interests. Then, based on the query vector and the FAISS index, a similarity search is performed in the vector database to obtain several initial interest points that form the initial candidate set.
[0036] In this embodiment, the preset cross-modal alignment model can adopt the SigLIP2 model. The SigLIP2 model, through a unified training objective, can generate high-quality image and text vector representations aligned in the same semantic space, thereby effectively capturing the core multimodal semantics of POIs. In this embodiment, the image and text vector representations of all POIs are pre-computed and stored in an efficient vector library, and a FAISS index is created. When the current recommendation request arrives, a query vector is generated based on the user's current interests, which can be extracted from the user's recent trajectory or explicit queries. Subsequently, similarity retrieval (such as an artificial neural network model ANN) is performed in the vector database using the query vector and the FAISS index to obtain the Top-N most relevant initial POIs, forming an initial candidate set. Furthermore, to meet the high throughput and low memory requirements during the recommendation process, this embodiment also introduces IVFPQ quantization indexing technology to achieve efficient approximate vector retrieval.
[0037] It is understandable that, after obtaining the initial candidate set, this embodiment refines the initial candidate set from different dimensions through two parallel processing paths. As shown in Figure 2, one path is spatiotemporal discovery and retrieval driven by multi-agent agents to obtain the spatiotemporal transition candidate set, and the other path is comment-driven preference reasoning and filtering to obtain the current user's preference feature information and preference candidate set.
[0038] Understandably, the multi-agent-driven spatiotemporal discovery and retrieval path focuses on mining deep spatiotemporal behavioral patterns from users' historical trajectories. As shown in Figure 4, this path, in constructing a spatiotemporal transition candidate set, includes, but is not limited to, the following steps: constructing a heterogeneous spatiotemporal graph based on historical trajectory data, where the nodes of the heterogeneous spatiotemporal graph are the initial points of interest in the initial candidate set, and the edges of the heterogeneous spatiotemporal graph have type information based on the relationships between the nodes; performing collaborative knowledge mining on the heterogeneous spatiotemporal graph to obtain fused spatiotemporal knowledge; and filtering the initial points of interest in the initial candidate set based on the fused spatiotemporal knowledge to obtain moment points of interest that conform to the current user's dynamic behavioral trends, forming a spatiotemporal transition candidate set.
[0039] Specifically, in the heterogeneous spatiotemporal graph G=(V,E), the nodes are points of interest, and the edges are assigned different types of information based on the relationships between the nodes, such as spatial proximity edges (geographically close), temporal rhythm edges (similar access time patterns), and mobility transition edges (sequentially appearing in the trajectory). In this embodiment, after obtaining the heterogeneous spatiotemporal graph, as shown in Figure 4, multiple specialized agents analyze the graph to obtain fused spatiotemporal knowledge. The specialized agents in this embodiment include, but are not limited to, mobility agents, spatial agents, temporal agents, and master agents. The mobility agent analyzes the transition edges in the heterogeneous spatiotemporal graph to identify the transition patterns between different initial points of interest for the current user. The spatial agent analyzes the spatial proximity edges in the heterogeneous spatiotemporal graph to obtain spatial knowledge corresponding to the current user's geographical activity range and regional preferences. The temporal agent analyzes the temporal rhythm edges in the heterogeneous spatiotemporal graph to discover the temporal knowledge of the current user's corresponding behavioral cycle patterns, such as the differences between weekdays and weekends. The master agent acts as a coordinator, performing cross-dimensional fusion and reasoning on the fragmented knowledge mined by the three agents mentioned above. Specifically, it performs cross-dimensional fusion and reasoning on knowledge of transfer patterns, spatial knowledge, and temporal knowledge to form structured fused spatiotemporal knowledge.
[0040] In this embodiment, after obtaining the fused spatiotemporal knowledge, the initial candidate set is subjected to "spatiotemporal transfer retrieval" or reordering using the fused spatiotemporal knowledge, thereby selecting the POIs that best match the user's current dynamic behavior trend to form a spatiotemporal transfer candidate set.
[0041] Understandably, the comment-driven preference inference and filtering path extracts the deep preference matching relationship between the current user and POIs from massive amounts of unstructured comment data. Due to the large size of the initial candidate set, this embodiment only considers the initial interest points corresponding to the current user under the current trajectory data. Specifically, this embodiment obtains the comment information and interaction record information corresponding to each initial interest point in the initial candidate set under the current trajectory data. As shown in Figure 5, the comment information and interaction record information are analyzed to obtain the current user's preference feature information, as well as the attribute feature information, including semantic summary information and sentiment attribute information, obtained through semantic analysis of the comment information. The fit between the preference feature information and attribute feature information corresponding to each initial candidate point in the initial candidate set is evaluated, and then the initial candidate points in the initial candidate set are filtered based on the evaluated fit to obtain the preference candidate set.
[0042] In the process of extracting preference feature information, this embodiment can analyze the current user's historical comment information and interaction records (e.g., likes, favorites) and then utilize the text understanding capabilities of the Large Language Model (LLM) to extract the current user's potential, fine-grained preference feature information, such as preference feature information categorized as "like" and "dislike" based on emotional difficulty. In the process of extracting attribute feature information, this embodiment can analyze the comment information on the initial candidate points of interest in the initial candidate set to extract key semantic summaries (e.g., "suitable for family dinners") and emotional attribute information ("advantages" and "disadvantages") to form attribute feature information. After obtaining the preference feature information and attribute feature information, this embodiment uses a preference analysis and semantic alignment mechanism, leveraging the reasoning capabilities of the Large Language Model (LLM), to evaluate the fit between the attribute feature information and preference feature information of each initial candidate POI. Based on this evaluation, the initial candidate set is filtered and reordered to obtain a highly personalized preference candidate set.
[0043] Understandably, after obtaining the spatiotemporal transition candidate set and preference candidate set, this embodiment dynamically constructs a prompt word template by combining current trajectory data, historical trajectory data, and preference feature information, guiding the large language model to complete the final recommendation task. Specifically, this embodiment inputs the prompt word template into the large language model to dynamically evaluate the relative importance of multi-source information in the prompt word template through the context learning ability of the large language model. Then, based on the evaluated relative importance, the initial interest points in the initial candidate set are reordered to obtain the target scene recommendation interest points in the target scene recommendation results. The recommendation explanation corresponding to the target scene recommendation interest points in the target scene recommendation results is then generated through natural language. The entire process of this embodiment does not involve any model parameter updates, thereby achieving intelligent recommendation without fine-tuning and effectively reducing the workload of the recommendation process.
[0044] In this embodiment of the application, as shown in Figure 2, after the prompt word template is input into the large language model, the large language model adopts the "exploration-reflection-knowledge consolidation" context strategy to generate target scene recommended interest points and corresponding recommended explanations. Among them, the "exploration-reflection-knowledge consolidation" context strategy involves: exploration using the powerful reasoning ability of LLM to generate reasoning trajectories; reflection extracting specific insights from successes and mistakes; and knowledge consolidation integrating these insights into structured context updates, thereby improving the accuracy of the recommendation results.
[0045] Taking the complete embodiment process shown in Figure 2 as an example, the method of this application embodiment is applied to several datasets for experimental testing. The specific implementation process includes, but is not limited to, the following steps: Step 1, Multimodal Recall Module: Utilize the SigLIP2 structure to extract joint representations from POI labels and images to generate an initial candidate set. This step integrates semantic retrieval and image matching capabilities to achieve cross-modal POI recall.
[0046] Step 2: Multi-Agent Driven Spatiotemporal Discovery and Retrieval: A spatiotemporal graph structure is constructed from historical trajectories. A multi-agent collaborative mechanism is introduced to explore the potential patterns of time-space transitions. After filtering the initial candidate set, a spatiotemporal transition candidate set is obtained. Each agent represents a different task dimension. Dynamic spatiotemporal migration features are extracted through a combination of graph traversal and context generation.
[0047] Step 3: Comment-driven preference reasoning and filtering: Combining current user behavior and comment data, the system analyzes the user's potential interest intentions and emotional tendencies, as well as the potential attribute characteristics (advantages and disadvantages) of POIs through a large language model. The initial candidate set is then filtered to obtain a subset of POIs that are more context-appropriate as the preference candidate set.
[0048] Step 4: Based on user trajectory, spatiotemporal transition candidate set, preference candidate set, and preference feature information, construct a multi-layered Prompt input to the large model. In the context engineering form of "exploration-method-knowledge consolidation", guide the large language model to generate the final target scene recommendation result.
[0049] In addition, this embodiment will compare the predicted target scenario recommendation results with the actual access data, and then extract new knowledge templates to strengthen the next Prompt construction and achieve closed-loop dynamic optimization.
[0050] As described above, this embodiment employs a framework comprised of three main threads working collaboratively: multimodal recall, spatiotemporally driven retrieval, and comment preference inference. This framework enables multi-dimensional, decoupled, in-depth analysis of user preferences. Furthermore, it replaces the traditional model fine-tuning paradigm with context engineering based on a large-scale language model. By dynamically injecting multi-source knowledge into the prompt, it guides the LLM (Language Modeling) to perform flexible, context-aware reasoning and decision-making, thereby achieving dynamic knowledge fusion through surface-level fine-tuning. Simultaneously, this example leverages the powerful context learning capabilities of LLM to capture dynamically changing user needs in real time without updating model parameters, achieving true instant personalization and improving the accuracy of recommendation results. Moreover, this embodiment utilizes the natural language generation advantages of LLM to provide clear and persuasive explanations for the final recommendation results.
[0051] Referring to Figure 6, this application embodiment also provides a scene recommendation device based on multimodal context-driven architecture. The device includes: a first module for acquiring a multimodal database of interest points and the current user's current recommendation request, current trajectory data, and historical trajectory data; a second module for constructing an initial candidate set based on the current recommendation request and the multimodal database of interest points, wherein the interest points in the initial candidate set are all related to the current user's current interests; a third module for performing spatiotemporal transition retrieval on the initial candidate set based on historical trajectory data to obtain a spatiotemporal transition candidate set; a fourth module for performing preference retrieval on the initial candidate set based on current trajectory data to obtain the current user's preference feature information and preference candidate set; a fifth module for constructing a prompt word template based on the current trajectory data, historical trajectory data, preference feature information, preference candidate set, and spatiotemporal transition candidate set; and a sixth module for inputting the prompt word template into a large language model to predict the current user's target scene interest points to obtain a target scene recommendation result.
[0052] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0053] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.
[0054] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0055] Please refer to Figure 7, which illustrates the hardware structure of an electronic device according to another embodiment. The electronic device includes: a processor 710, which can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, for executing related programs to implement the technical solutions provided in the embodiments of this application; and a memory 720, which can be implemented using a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM), etc. The memory 720 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710. The input / output interface 730 is used to implement information input and output. The communication interface 740 is used to realize communication interaction between this device and other devices. Communication can be realized by wired means (such as USB, network cable, etc.) or by wireless means (such as mobile network, WIFI, Bluetooth, etc.). The bus 750 transmits information between various components of the device (such as processor 710, memory 720, input / output interface 730 and communication interface 740). The processor 710, memory 720, input / output interface 730 and communication interface 740 realize communication connection between each other within the device through the bus 750.
[0056] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method.
[0057] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.
[0058] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.
[0059] It is understood that the content of the above method embodiments is applicable to the embodiments of this program product. The specific functions implemented by the embodiments of this program product are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.
[0060] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.
[0061] This application provides a scene recommendation method and related device based on multimodal context-driven approaches. After constructing an initial candidate set based on the current recommendation request and a multimodal database of points of interest, a spatiotemporal transition candidate set is obtained by performing spatiotemporal transition retrieval on the initial candidate set based on historical trajectory data. Similarly, preference retrieval is performed on the initial candidate set based on current trajectory data to obtain the current user's preference feature information and preference candidate set. Then, a prompt word template is constructed based on the current trajectory data, historical trajectory data, preference feature information, preference candidate set, and spatiotemporal transition candidate set, ensuring that the information in the prompt word template corresponds to the current user's current scene. Finally, the prompt word template is input into a large language model to predict the current user's target scene points of interest, thus obtaining the target scene recommendation result. This leverages the context learning capability of the large language model to improve the accuracy of the prediction results, thereby improving the accuracy of the recommendation results and reducing the workload of the recommendation process.
[0062] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.
[0063] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.
[0064] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.
[0065] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.
[0066] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0067] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.
[0068] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0069] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0070] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0071] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0072] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.
Claims
1. A scene recommendation method based on multimodal context-driven approach, characterized in that, The method includes the following steps: acquiring a multimodal database of points of interest and the current user's current recommendation request, current trajectory data, and historical trajectory data; constructing an initial candidate set based on the current recommendation request and the multimodal database of points of interest, wherein all points of interest in the initial candidate set are related to the current user's current interests; performing spatiotemporal transition retrieval on the initial candidate set based on the historical trajectory data to obtain a spatiotemporal transition candidate set; performing preference retrieval on the initial candidate set based on the current trajectory data to obtain the current user's preference feature information and preference candidate set; constructing a prompt word template based on the current trajectory data, the historical trajectory data, the preference feature information, the preference candidate set, and the spatiotemporal transition candidate set; and inputting the prompt word template into a large language model to predict the current user's target scene points of interest to obtain a target scene recommendation result.
2. The method according to claim 1, characterized in that, The step of constructing an initial candidate set based on the current recommendation request and the multimodal interest database includes: jointly encoding the image information and label text information of each interest point in the multimodal interest database using a preset cross-modal alignment model to obtain image and text vector representations; storing the image and text vector representations corresponding to all interest points in a vector database and creating a FAISS index for the vector database; when the current recommendation request is received, generating a query vector based on the current user's current interests; and performing a similarity search in the vector database based on the query vector and the FAISS index to obtain several initial interest points that form the initial candidate set.
3. The method according to claim 2, characterized in that, The step of performing spatiotemporal transition retrieval on the initial candidate set based on the historical trajectory data to obtain a spatiotemporal transition candidate set includes: constructing a heterogeneous spatiotemporal graph based on the historical trajectory data, wherein the nodes of the heterogeneous spatiotemporal graph are the initial points of interest in the initial candidate set, and the edges of the heterogeneous spatiotemporal graph are provided with type information based on the relationship between the nodes; performing collaborative knowledge mining on the heterogeneous spatiotemporal graph to obtain fused spatiotemporal knowledge; and filtering the initial points of interest in the initial candidate set based on the fused spatiotemporal knowledge to obtain the moment points of interest that conform to the current user dynamic behavior trend, which constitute the spatiotemporal transition candidate set.
4. The method according to claim 3, characterized in that, The collaborative knowledge mining of the heterogeneous spatiotemporal graph to obtain fused spatiotemporal knowledge includes: analyzing the transitional edges in the heterogeneous spatiotemporal graph to obtain the transitional pattern knowledge of the current user between different initial points of interest; analyzing the spatial proximity edges in the heterogeneous spatiotemporal graph to obtain the spatial knowledge corresponding to the geographical activity range and regional preferences of the current user; analyzing the temporal rhythm edges in the heterogeneous spatiotemporal graph to obtain the temporal knowledge corresponding to the behavioral cycle pattern of the current user; and performing cross-dimensional fusion and reasoning on the transitional pattern knowledge, the spatial knowledge, and the temporal knowledge to obtain structured fused spatiotemporal knowledge.
5. The method according to claim 1, characterized in that, The step of performing preference retrieval on the initial candidate set based on the current trajectory data to obtain the current user's preference feature information and preference candidate set includes: obtaining comment information and interaction record information corresponding to each initial point of interest in the initial candidate set under the current trajectory data; analyzing the comment information and interaction record information to obtain the current user's preference feature information; performing semantic analysis on the comment information to obtain attribute feature information, the attribute feature information including semantic summary information and sentiment attribute information; evaluating the fit between the preference feature information and the attribute feature information corresponding to each initial candidate point in the initial candidate set; and filtering the initial candidate points in the initial candidate set according to the evaluated fit to obtain the preference candidate set.
6. The method according to claim 1, characterized in that, The step of inputting the prompt word template into a large language model to predict the target scene interest points of the current user and obtain the target scene recommendation result includes: inputting the prompt word template into the large language model, dynamically evaluating the relative importance of multi-source information in the prompt word template through the context learning ability of the large language model; reordering the initial interest points in the initial candidate set according to the evaluated relative importance to obtain the target scene recommended interest points in the target scene recommendation result; and generating a recommendation explanation corresponding to the target scene recommended interest points in the target scene recommendation result through natural language.
7. The method according to claim 1, characterized in that, The method further includes the following steps: obtaining comparison data between the target scene recommendation results and the target scene actual access data; and updating the prompt word template based on the comparison data.
8. A scene recommendation device based on multimodal context-driven approach, characterized in that, The device includes: a first module for acquiring a multimodal database of points of interest and the current user's current recommendation request, current trajectory data, and historical trajectory data; a second module for constructing an initial candidate set based on the current recommendation request and the multimodal database of points of interest, wherein the points of interest in the initial candidate set are all related to the current user's current interests; a third module for performing spatiotemporal transition retrieval on the initial candidate set based on the historical trajectory data to obtain a spatiotemporal transition candidate set; a fourth module for performing preference retrieval on the initial candidate set based on the current trajectory data to obtain the current user's preference feature information and preference candidate set; a fifth module for constructing a prompt word template based on the current trajectory data, the historical trajectory data, the preference feature information, the preference candidate set, and the spatiotemporal transition candidate set; and a sixth module for inputting the prompt word template into a large language model to predict the current user's target scene points of interest to obtain a target scene recommendation result.
9. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; when said at least one program is executed by said at least one processor, said at least one processor implements the method as described in any one of claims 1 to 7.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.