A cross-domain recommendation system between communities based on multi-modal large model alignment

Abstract

The application discloses a community cross-domain recommendation system based on multi-modal large model alignment, and relates to the technical field of artificial intelligence recommendation. The system includes multi-modal alignment, prompt construction and teacher-student alignment modules. First, the multi-modal large model is used to convert the heterogeneous images and text data of interest points into unified structured semantic labels, realizing cross-modal feature alignment. Second, the user's auxiliary travel history is introduced and combined with the thinking chain reasoning mechanism to accurately decouple the preference differences of users in the resident mode and the tourist mode. Finally, a two-stage strategy of supervised fine-tuning and direct preference optimization is adopted to efficiently transfer the reasoning ability of the large teacher model to the lightweight student model. The application effectively solves the semantic gap and preference deviation problem in off-site recommendation, realizes real-time recommendation within seconds while ensuring high accuracy, and is suitable for online travel services and location service scenarios.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and Internet information recommendation technology, specifically to a cross-regional recommendation re-ranking system that utilizes a multimodal large model (MLLM) for heterogeneous data semantic alignment, decouples user scenario preferences through a thought chain (CoT) reasoning mechanism, and adopts a teacher-student alignment strategy to achieve efficient deployment. Background Technology

[0002] With the popularization of mobile internet technology and the recovery of the global tourism economy, location-based services (LBS) are playing an increasingly important role in people's daily lives. Among them, out-of-town recommendation (OOT), as one of the core application scenarios of LBS, aims to solve the problem of how users can quickly obtain suggestions for restaurants, attractions, or entertainment facilities (POIs) that match their personal tastes when they leave their usual city and enter a completely unfamiliar geographical environment.

[0003] Compared to traditional local recommendation, cross-regional recommendation faces more complex data challenges, mainly in terms of data sparsity and the cold start problem. To address these issues, academia and industry typically employ cross-domain recommendation (CDR) technology, attempting to migrate the rich behavioral data accumulated by users in the source domain (i.e., their home city) to the target domain (i.e., their tourist city) to aid in preference prediction in the target domain.

[0004] However, existing off-site recommendation technologies have the following significant technical defects and bottlenecks in practical applications:

[0005] First, existing multimodal fusion techniques struggle to achieve deep semantic cross-modal reasoning. In recommender systems, POI features are often multimodal and heterogeneous. For example, a restaurant photo (visual modality) might reflect a "retro industrial" style, while a user's textual review (textual modality) might describe their experience using adjectives like "nostalgic" or "noisy." Existing mainstream recommender models (such as MMGCN) typically employ "late fusion" or simple vector concatenation strategies, mathematically concatenating image feature vectors with text embedding vectors. This approach fails to establish an explicit semantic connection between image content and text description, preventing the model from performing logical reasoning like a human (e.g., "The user likes the retro style in the picture, which matches the nostalgic description in the review"). This "semantic gap" severely limits the recommender system's ability to understand fine-grained POI features.

[0006] Second, it ignores the phenomenon of "preference deviation" in different geographical scenarios. Most existing cross-domain recommendation algorithms are based on a strong assumption: user preferences are static and consistent. In other words, they assume that user behavior patterns in their home city can be directly transferred to tourist cities. However, psychological and behavioral studies show that user behavioral intentions differ significantly in different scenarios. In "home city mode," influenced by work and life rhythms, users tend to choose places with convenient commuting, high cost-effectiveness, or business attributes (such as fast food restaurants and business cafes); while in "tourist mode," user intentions shift to exploration, leisure, and experience, leading to a preference for places with local cultural characteristics, scenic advantages, or high ratings (such as seaside restaurants and historical sites). Existing technical solutions often fail to effectively distinguish between these two modes, resulting in models trained directly on home city history recommending a large number of "fast food" or "chain stores" that do not match the tourist's mindset in different locations, greatly reducing the user experience.

[0007] Third, recommendation schemes based on large models struggle to balance inference performance and response latency. In recent years, large language models (LLMs) have been attempted for inclusion in the re-ranking stage of recommender systems due to their powerful knowledge reserves and logical reasoning capabilities. However, general-purpose large models (such as GPT-4 and DeepSeek) have enormous parameter counts (typically in the hundreds of billions), requiring significant amounts of GPU memory and computation time for a single inference, resulting in response latency often reaching seconds or even minutes. For online recommender systems requiring millisecond-level response times, such high latency is unacceptable. Therefore, how to retain the powerful inference capabilities of large models while compressing them into lightweight models that can run online in real-time is a pressing technological challenge.

[0008] In summary, existing technologies lack a comprehensive solution that can simultaneously address multimodal semantic alignment, user scenario preference decoupling, and high-efficiency online inference. Summary of the Invention

[0009] To address the aforementioned problems in existing technologies, this invention provides a cross-regional recommendation re-ranking system (DiMA) based on multimodal large model alignment. This system solves the cross-modal understanding challenge by constructing a unified semantic label space, addresses preference shift issues by utilizing a thought chain mechanism and auxiliary travel history, and resolves online inference efficiency issues through two-stage teacher-student alignment training.

[0010] The main technical problems to be solved by this invention include:

[0011] (1) How to transform heterogeneous unstructured data (images, comment text) into unified structured semantic features that are machine-understandable and reasonable;

[0012] (2) How to accurately extract users' unique preferences in "guest mode" from their mixed historical behaviors;

[0013] (3) How to construct an efficient model training and distillation framework so that lightweight models can achieve reordering accuracy that surpasses that of large teacher models.

[0014] To solve the above-mentioned technical problems, the technical solution adopted by the present invention mainly includes the following three core processing modules:

[0015] 1. Multi-Modal Alignment Module

[0016] This module serves as the system's data preprocessing engine, configured to map raw POI multimodal data to a unified semantic label space.

[0017] (1) Image semantic translation unit: integrates pre-trained multimodal large model (MLLM, such as the InternVL series) to perform visual content understanding on multiple display images of each POI and generate natural language description text containing details such as scene, object, atmosphere, etc.

[0018] (2) Structured Tag Extraction Unit: Utilizing a large-scale language model (LLM, such as DeepSeek-V3) with instruction-following capabilities, the unit takes image description text and user history comments as input and extracts two types of key tags through preset prompts. Entity Tags: These describe the core attributes of the POI, such as "coffee," "seafood," "terrace," and "live music." Emotion Tags: These describe the user's sensory experience, such as "comfortable," "crowded," "expensive," and "attentive service."

[0019] (3) Tag normalization unit: The built-in natural language processing algorithm performs lemmatization and stemming on the extracted tags to eliminate differences in singular and plural forms, tenses, etc., and generate the final unified semantic tag set.

[0020] 2. Prompt Construction Module

[0021] This module is used to transform multi-source heterogeneous data into structured thought chain hints that can be understood by large models.

[0022] (1) Multi-source historical aggregation unit: respectively obtain the user's "historical interaction sequence of the city where he resides" (representing daily preferences) and "historical interaction sequence of the auxiliary tourist city" (representing historical tourist preferences), as well as the "candidate POI list" of the current target city.

[0023] (2) Collaborative signal calculation unit: Based on the interaction matrix of all users, calculate the Jaccard similarity between the target user and other users in the resident domain and auxiliary domain, and then calculate the "category-level collaborative filtering score" and "venue-level collaborative filtering score" for each POI in the candidate list, introducing collective intelligence as an auxiliary decision-making factor.

[0024] (3) CoT Template Generation Unit: The above historical sequence, metadata (name, location) of candidate POIs, normalized semantic labels, and collaborative filtering scores are filled into the preset CoT template. This template forces the model to perform the following inference steps:

[0025] Step 1 (User Profile Synthesis): Compare the user's regular travel history with the supplementary travel history, analyze the differences, and synthesize the user's preference hypothesis under the "tourist mode" (e.g., "Frugal in daily life, but prefers high-end experiences when traveling").

[0026] Step 2 (Multidimensional Candidate Evaluation): Based on the synthetic preference assumptions, and combining labels and collaboration scores, each candidate POI is evaluated individually.

[0027] Step 3 (Final Sorting Decision): Output the reordered list of IDs.

[0028] 3. Teacher-Student Alignment Module

[0029] This module is used for offline training and capability transfer of models.

[0030] (1) Teacher reasoning data generation unit: The above-constructed prompts are processed using a high-performance teacher model (such as DeepSeek-R1) to generate a high-quality dataset containing the complete reasoning process text and the final ranking results.

[0031] (2) First stage supervised fine-tuning (SFT) unit: The lightweight student model (such as Qwen3-4B) is fine-tuned with all parameters using the dataset generated by the teacher model. The optimization goal is to minimize the cross-entropy loss generated by autoregression, so that the student model can learn the teacher's thought chain reasoning logic.

[0032] (3) Second stage Direct Preference Optimization (DPO) unit: Construct preference triples .in, For input prompts, The winning response is an ideal ranking constructed based on real user click data (Ground Truth). (Failed responses) are the original rankings generated by the teacher model. By maximizing the log-likelihood difference between winning and failing responses, the student model is forced to correct the teacher model's incorrect judgments, thereby achieving performance superiority.

[0033] 4. The beneficial effects of the present invention:

[0034] (1) Breaking the cross-modal semantic gap: By mapping images and text to structured labels in a unified manner, this system enables the recommendation model to reason based on explicit semantic symbols, which greatly improves the depth and accuracy of understanding POI content.

[0035] (2) It achieves precise decoupling of user preferences: It innovatively introduces "auxiliary tourism history" as a reference system, and with the help of thought chain reasoning, it effectively solves the long-standing problem of "resident-tourist" preference confusion in the field of cross-regional recommendation, and significantly improves recommendation satisfaction in cross-regional scenarios.

[0036] (3) Balancing high performance and low latency: Through a teacher-student alignment strategy, the complex reasoning capabilities of the large model were successfully distilled into a lightweight small model. Experimental data show that the optimized student model is more than 30 times faster than the teacher model in reasoning, fully meeting the requirements for deployment in industrial-grade real-time systems.

[0037] (4) It has strong zero-shot generalization ability: The model trained by this system does not depend on a specific candidate generator and can be directly applied to different cities and data sources, with strong robustness and reusability. Attached Figure Description

[0038] Figure 1 is a schematic diagram illustrating the principle of preference shift in cross-regional recommendation proposed in this invention. The figure shows the comparative effect of traditional models relying solely on local history, leading to recommendation bias, while this invention introduces auxiliary travel history to decouple preferences.

[0039] Figure 2 is a flowchart of the overall algorithm architecture of the DiMA system provided by this invention. The figure shows in detail the entire data flow from raw data input, through multimodal alignment and prompt construction, to the teacher-student alignment training module.

[0040] Figure 3 is a bar chart comparing the performance of student models of different sizes in this invention on the Foursquare dataset. The chart shows the improvement trend of recommendation metrics (NDCG@5, MAP@5) as the number of model parameters increases, and the comparative advantages compared to the baseline model.

[0041] Figure 4 is a flowchart of a practical example of cross-modal reasoning using the system of this invention. This figure illustrates the specific logical path of how the system extracts visual labels from images and matches them with text labels in comments, thereby achieving cross-modal recommendation. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0043] [Example 1: Semantic Alignment Process for Multimodal Data]

[0044] This embodiment describes in detail the working process of the multimodal alignment module. Assume the system is processing a restaurant called "Rubio's Coastal Grill" located in Los Angeles from the Foursquare dataset.

[0045] Image Processing: The system first reads 10 high-resolution images of the restaurant from the database. These images are then fed into the InternVL-2.5-8B-MPO multimodal model. The model output is described as follows: "A bright interior photo with several tacos filled with fried fish and cabbage on the table, against a blue ocean-themed background."

[0046] Tag Extraction: The system inputs the above descriptive text along with the restaurant's 50 most recent user reviews (such as "Greatfish tacos and lively atmosphere") into the DeepSeek-V3 model. The prompts require the model to extract "entities" and "sentiments."

[0047] Output results: The model outputs entity labels {taco, fish, seafood, ocean_decor} and sentiment labels {lively, great, casual}.

[0048] Normalization: The system processes the above tags using the NLTK library, for example, restoring "tacos" to "taco" and simplifying "ocean_decor" to "ocean". The resulting unified tag set is stored in the knowledge base for subsequent reasoning. This process greatly compresses the original data while preserving the core semantics.

[0049] [Example 2: User Preference Decoupling Reasoning Based on Mind Chain]

[0050] This example demonstrates how the prompting module constructs input to address preference bias. Assume the target user is accessing Los Angeles (the target domain).

[0051] Historical search: The system retrieved 15 interactions of the user in their place of residence (such as New York), mostly involving chain fast food such as "Starbucks" and "Subway"; it also retrieved information that the user had visited San Francisco (auxiliary domain) and visited local attractions such as "Fisherman's Wharf Seafood" and "Cliff House".

[0052] Hint: The system is built as follows (CoT). Hint:

[0053] Input information: Currently residing in New York (mainly fast food); previously visited San Francisco (mainly for seafood and sightseeing); currently in Los Angeles.

[0054] Instruction: Analyze the differences between this user's home location preferences and travel preferences. Based on this analysis, reorder the candidate list for Los Angeles.

[0055] Reasoning process (output of the simulated teacher model):

[0056] Step 1 Analysis: "Users prioritize efficiency and standardization (fast food) in their place of residence, but when traveling (as seen in San Francisco), they clearly tend to seek out local specialties and natural landscapes. Therefore, in Los Angeles, POIs with 'specialty' or 'landscape' attributes should be prioritized, while the weighting of chain stores should be reduced."

[0057] Step 2 Evaluation: "Candidate A (a chain burger restaurant): low tag match. Candidate B (Nopalito restaurant): tags include 'taco' (specialty food) and 'lively' (lively atmosphere), similar to San Francisco travel preferences, and has a high collaborative filtering score."

[0058] Step 3: Sort: "Rank candidate B before candidate A".

[0059] [Example 3: Two-stage teacher-student alignment training and online services]

[0060] This embodiment describes how to obtain a final deployable lightweight model through training.

[0061] Data preparation: The system uses DeepSeek-R1 (teacher model) to process 10,000 requests similar to those in Example 2 to generate training data containing complete inference text.

[0062] SFT Phase (Supervised Fine-Tuning): Initialize a Qwen3-4B (student model). Fine-tune all parameters using the aforementioned data, optimizing the objective function to minimize the negative log-likelihood loss of the generated inference text. After 3 epochs of training, the student model perfectly replicates the teacher's "analysis-evaluation-ranking" logic.

[0063] DPO Phase (Direct Preference Optimization): To outperform the teacher model, the system incorporates real click data. For the same query, if the teacher model recommends a list... However, if the user actually clicks on the 5th item in the list, the system will construct a "winning response". (List the items that have been elevated in item 5), and set the teacher's original output to "failure response". The student model was fine-tuned using the DPO loss function.

[0064] Online service: During the actual inference phase, the system only loads the pre-trained Qwen3-4B student model. When a user makes a request, the system no longer needs to call the expensive teacher model. Instead, the student model directly outputs the recommendation result containing the reasoning basis within 10 seconds (based on RTX3090 testing), achieving a perfect balance between high accuracy and low latency.

Claims

1. A cross-domain recommendation system for inter-community communication based on multimodal large model alignment, characterized by: The multimodal alignment module is used to transform heterogeneous point of interest (POI) data into unified structured semantic labels using multimodal large models and large language models to achieve cross-modal reasoning; The prompt building module is used to integrate user history, candidate interest list and semantic tags, and embed them into prompt templates that contain thought chain (CoT) reasoning steps; The teacher-student alignment module is used to guide the teacher model in generating inference trajectories and sorting lists, and transfers this capability to the lightweight student model through a two-stage alignment strategy.

2. The cross-regional recommendation re-ranking system according to claim 1, wherein the multimodal alignment module comprises: The image-to-text submodule uses a multimodal large model to convert POI images into text descriptions; The tag extraction submodule extracts objective entity tags from text descriptions and user comments, and extracts subjective sentiment tags from user comments; The tag normalization submodule standardizes the entity tags and sentiment tags through stemming and word form restoration to generate a unified semantic space.

3. In the cross-regional recommendation and re-ranking system according to claim 1, the prompt construction module utilizes the user's permanent residence history in the permanent residence city and the auxiliary travel history in the third-party city to decouple the user's permanent residence preference pattern and tourist preference pattern through comparative analysis, so as to solve the preference shift problem.

4. The cross-regional recommendation re-ranking system according to claim 1 or 3, wherein the thought chain reasoning step includes: Step 1 (User Analysis): Force the model to compare residency history and travel history to form hypotheses about users' preferences when they are tourists; Step 2 (Candidate Evaluation): Integrate semantic label signals and collaborative filtering scores to evaluate each candidate interest point based on the stated preference hypothesis; Step 3 (Reordering): Generate the final recommendation list based on the comprehensive evaluation results.

5. In the cross-regional recommendation re-ranking system according to claim 1, the teacher-student alignment module performs the following two-stage alignment: Phase 1 (SFT): Through supervised fine-tuning, the student model minimizes the autoregressive loss between the student model and the teacher model's complete inference output, thereby replicating the teacher model's thought process. Phase 2 (DPO): Through direct preference optimization, the student model is optimized using the constructed preference triples to align it with the user's true choices.

6. In the cross-regional recommendation re-ranking system according to claim 5, the winner ranking in the preference triple is constructed based on real interaction data, while the loser ranking is the original ranking list generated by the teacher model, thereby guiding the model performance to exceed the upper limit of the teacher model.

7. The cross-regional recommendation re-ranking system according to any one of claims 1 to 6, wherein the prompt construction module further includes a collaborative signal calculation submodule, used to calculate the shared interaction history of the target user and other users in multiple cities to obtain a collaborative filtering score based on category level and venue level, and use it as input features to participate in the reasoning process.

8. The cross-regional recommendation re-ranking system according to any one of claims 1 to 6, wherein during the inference phase, the system only calls a trained lightweight student model and generates a response containing inference text and a JSON-formatted sorted list through one-step autoregression to reduce the inference latency of real-time recommendations.

Images