Big language model dialogue recommendation method based on multi-modal geographic information fusion and context modulation

Abstract

The invention discloses a big language model dialogue recommendation method based on multi-modal geographic information fusion and context modulation, which comprises the following steps of: firstly, coding and fusing a text address, geographic coordinates and numerical attributes to obtain a geographic context matrix; then intention modeling is carried out to generate user intention representation, then context feature modulation is utilized, dynamic modulation is carried out on the user intention representation by utilizing an asymmetric fusion mechanism, a context guiding signal containing geographical constraints is generated, and finally, the context guiding signal is spliced before the user intention representation. The big language model input to the parameter freezing obtains user preference vectors and calculates recommendation scores with the candidate positions, the candidate positions are sorted according to the recommendation scores, and the candidate positions with high scores are recommended to the user. According to the method, on the premise that internal parameters of the large language model are not changed, the geospatial reasoning ability is effectively given to the large language model, and the accuracy and interpretability of dialogue recommendation are remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of dialogue recommendation, and more specifically relates to a large language model dialogue recommendation method based on multi-modal geographic information fusion and context modulation. BACKGROUND

[0002] Conversational recommender systems (CRS) dynamically capture user preferences through multi-round natural language interaction, and have wide application prospects in urban tourism, local life services and other fields. With the development of large language models (LLM), using the powerful semantic understanding ability of large language models for recommendation has become a research hotspot.

[0003] However, in scenarios involving deep geographic perception, existing dialogue recommendation methods face two major challenges:

[0004] 1. Heterogeneity and representation gap of geographic information

[0005] Geographic data usually contains unstructured text addresses (such as "Beijing Haidian District..."), nonlinear latitude and longitude coordinates, and structured numerical attributes (such as ratings, prices). Existing large language models can usually only process text and are difficult to directly understand the geometric spatial relationship in latitude and longitude coordinates, resulting in a "spatial perception blind area".

[0006] 2. Symmetry defect of fusion mechanism

[0007] In dialogue recommendation methods, the user's immediate dialogue intent (User Intent) is the dominant signal for decision-making, while the user's historical geographic context (Geographic Context) should be used as a background constraint. Existing multi-modal recommendation methods usually use simple vector concatenation or summation for "symmetric fusion". This approach ignores the primary and secondary dependency relationship and is prone to cause the key user intent signal to be overwhelmed by the complex geographic history information, i.e., the "primary and secondary inversion" problem, thereby reducing the accuracy of dialogue recommendation.

[0008] Therefore, there is an urgent need for a dialogue recommendation method that can efficiently fuse heterogeneous geographic information without compromising the general capabilities of large language models and achieve asymmetric context modulation. SUMMARY

[0009] The present application aims to overcome the shortcomings of the prior art and provide a large language model dialogue recommendation method based on multi-modal geographic information fusion and context modulation, to solve the problem of heterogeneous geographic information being difficult to uniformly represent and direct fusion causing user intent to be disturbed by geographic background noise, and to improve the accuracy of dialogue recommendation.

[0010] To achieve the above-mentioned object of the application, the application is based on a multimodal geographic information fusion and context modulation large language model dialogue recommendation method, characterized by comprising the following steps:

[0011] (1) Hierarchical geographic coding

[0012] Obtain the location sequence in the user interaction history, encode the text address, geographic coordinates and numerical attributes of each location in the location sequence respectively, and fuse to generate a location representation, and all location representations are arranged in rows to obtain a geographic context matrix;

[0013] (2) User intent modeling

[0014] Obtain the current dialogue history text, extract semantic features using a pre-trained language model, and generate a user intent representation;

[0015] (3) Context feature modulation

[0016] The geographic context matrix is used as a conditional variable, and an asymmetric fusion mechanism is used to dynamically modulate the user intent representation to generate a context guide signal containing geographic constraints;

[0017] (4) Large language model recommendation based on guidance

[0018] The context guide signal is regarded as a soft prompt and spliced before the user intent representation, and input into a parameter frozen large language model to obtain a user preference vector and calculate a recommendation score with a candidate location embedding, and the candidate locations are sorted according to the recommended score, and the locations with high scores are recommended to the user.

[0019] The object of the application is achieved in this way.

[0020] The multimodal geographic information fusion and context modulation large language model dialogue recommendation method of the application first encodes and fuses text addresses, geographic coordinates and numerical attributes to obtain a geographic context matrix, then performs intent modeling to generate a user intent representation, and then uses context feature modulation to dynamically modulate the user intent representation using an asymmetric fusion mechanism to generate a context guide signal containing geographic constraints, and finally splices the context guide signal before the user intent representation, inputs it into a parameter frozen large language model to obtain a user preference vector and calculates a recommendation score with a candidate location, and the candidate locations are sorted according to the recommended score, and the locations with high scores are recommended to the user. Without changing the internal parameters of the large language model, the application effectively gives the large language model the ability to reason in geographic space, significantly improving the accuracy and explainability of dialogue recommendation.

[0021] Compared with the prior art, the application has the following beneficial effects:

[0022] 1、Optimized primary and secondary signal fusion

[0023] The context feature modulation mechanism with an asymmetric fusion mechanism ensures that geographic information is only used as a "condition" to adjust user intent, avoids the drowning of intent signals caused by direct splicing, and significantly improves the accuracy of recommendations.

[0024] 2、Efficient and low cost

[0025] The present application only needs to train an external lightweight encoder and a modulation module, without full parameter fine-tuning of a large language model, greatly reducing the training cost, while retaining the general language ability of the large language model, avoiding catastrophic forgetting. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a structural diagram of a dialogue recommendation system designed by the dialogue recommendation method of the large language model based on multi-modal geographic information fusion and context modulation of the present application.

[0027] Figure 2 is a flowchart of one specific embodiment of the dialogue recommendation method of the large language model based on multi-modal geographic information fusion and context modulation of the present application. DETAILED DESCRIPTION

[0028] The specific embodiments of the present application will be described below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present application. It should be particularly noted that in the following description, when the detailed description of known functions and designs may obscure the main content of the present application, these descriptions will be omitted here.

[0029] Figure 1 is a structural diagram of a dialogue recommendation system designed by the dialogue recommendation method of the large language model based on multi-modal geographic information fusion and context modulation of the present application.

[0030] In this embodiment, as Figure 1As shown, the dialogue recommendation system designed based on the application includes three parts: hierarchical geographic encoder, user intent modeling, and geographic attention recommendation. Among them, the hierarchical geographic encoder is used to encode the text address, geographic coordinates and numerical attributes of each location in the location sequence respectively, and fuse to generate a location representation. All location representations are arranged in rows to obtain a geographic context matrix. The intent modeling is used to obtain the current dialogue history text, extract semantic features using a pre-trained language model, and generate a user intent representation. The geographic attention recommendation includes context feature modulation and guided large language model recommendation generation. That is, the geographic context matrix is taken as a conditional variable, the user intent representation is dynamically modulated using an asymmetric fusion mechanism, a context guide signal containing geographic constraints is generated, the context guide signal is spliced before the user intent representation, and input into a parameter frozen large language model to obtain a user preference vector and calculate a recommendation score with a candidate location embedding. According to the score of the recommendation, the candidate locations are sorted, and the locations with high scores are recommended to the user.

[0031] Figure 2 is a specific implementation flowchart of the application based on multi-modal geographic information fusion and context modulation large language model dialogue recommendation method.

[0032] In this embodiment, as shown in Figure 1 , 2 the application based on multi-modal geographic information fusion and context modulation large language model dialogue recommendation method includes the following steps:

[0033] Step S1: hierarchical geographic encoding

[0034] Obtain the location sequence in the user interaction history, encode the text address, geographic coordinates and numerical attributes of each location in the location sequence respectively, and fuse to generate a location representation. All location representations are arranged in rows to obtain a geographic context matrix.

[0035] In this embodiment, for each location in the user interaction history , the application encodes the text address, coordinate geography and numerical attributes respectively. The specific encoding is as follows:

[0036] Step S1.1: Address Encoding

[0037] The text address is divided into multiple spatial units according to the administrative level or semantic structure, the feature vectors of each spatial unit are extracted using a text encoder, a learnable query vector is introduced, the attention weights of each spatial unit are calculated, and the feature vectors are weighted and summed based on the weights to obtain a unified address representation.

[0038] Address texts have a hierarchical structure, like city-district-street. In this embodiment, the text address is first split into spatial units where denotes the sequence number of a position in the sequence of positions, denotes the text address in the position;

[0039] Using a pre-trained text encoder, in this embodiment RoBERTa, a feature vector is extracted for each spatial unit

[0040]

[0041] where denotes the sequence number of a spatial unit, denotes average pooling, denotes the RoBERTa pre-trained text encoder.

[0042] To aggregate these spatial units, a learnable query vector is introduced, and the attention weights are computed:

[0043]

[0044] where denotes transposition, denotes the sequence number of a spatial unit;

[0045] The final address representation for the -th position is a weighted sum:

[0046]

[0047] Step S1.2: Coordinates Encoding

[0048] In this embodiment, the geographic coordinates are latitude-longitude spherical coordinates. To address the issue that latitude-longitude spherical coordinates are not suitable for local spatial reasoning, in this embodiment, a strategy of “projection + multi-scale encoding” is adopted:

[0049] A map projection algorithm is used to convert the latitude-longitude coordinates of a position into local planar Cartesian coordinates;

[0050] A multi-scale sinusoidal function is used to encode the local planar Cartesian coordinates to generate position features at different spatial frequencies;

[0051] The position features at different scales are concatenated to obtain the multi-scale coordinate representation of the -th position .

[0052] Specifically,

[0053] First, the longitude and latitude of the location are converted into local plane coordinates by using Lambert Conformal Conic (LCC) projection:

[0054] The projection constants and are calculated:

[0055]

[0056]

[0057] where are the first and second standard latitudes, respectively;

[0058] The polar coordinates are calculated:

[0059]

[0060]

[0061] where is the reference longitude;

[0062] The local plane coordinates are converted into Cartesian coordinates:

[0063]

[0064] where is the radius at the reference dimension;

[0065] Then, the local plane coordinates are subjected to multi-scale sinusoidal encoding, and for each scale , the encoding formula is:

[0066]

[0067]

[0068] where represents the total number of frequency dimensions at each scale, and the frequency is defined according to the maximum radius :

[0069]

[0070] Finalize all scales of multi-scale sinusoidal encoding Concatenate to get coordinate representation .

[0071] Step S1.3: Numerical Attribute Encoding

[0072] In this embodiment, for numerical attributes including domain, rating, price, etc., the present application adopts a numerical encoder based on Transformer for processing.

[0073] First, construct an input embedding sequence : map the value of the th numerical attribute to a numerical embedding vector , The total number of numerical attributes, and respectively superimpose a learnable type embedding vector to distinguish attribute categories, while introducing a classification marker embedding and its type embedding at the beginning of the sequence to aggregate global information, whose calculation formula is as follows:

[0074]

[0075] Next, input the constructed input embedding sequence into the Transformer encoder to capture the deep dependency between attributes, and extract the hidden state corresponding to the CLS marker in the output layer as the unified feature representation of the numerical attribute at this position , whose calculation formula is as follows:

[0076]

[0077] Step S1.4: Fusing

[0078] Concatenate the above three representations and fuse them through a multilayer perceptron (MLP) to generate a position representation :

[0079]

[0080] Arrange the position representations of all positions in the user interaction history in rows to obtain a geographical context matrix .

[0081] Step S2: User Intent Modeling

[0082] Obtain the current dialogue history text, extract semantic features using a pre-trained language model, and generate a user intent representation.

[0083] Given the current dialogue history , use a pre-trained language model to extract a user intent representation :

[0084]

[0085] Step S3: Contextual Feature Modulation (CFM)

[0086] Use the geographical context matrix as a conditional variable and use an asymmetric fusion mechanism to dynamically modulate the user intent representation, generating a context-guided signal containing geographical constraints.

[0087] This is a core innovation of the present application, containing two sub-steps.

[0088] Step S3.1: Content Alignment

[0089] Use the Cross-Attention mechanism to find the most relevant geographical history to the user intent representation , i.e. attention , to obtain the aligned representation :

[0090]

[0091]

[0092] In the specific implementation process, as shown in Figure 1 , all linear mapping is performed;

[0093] Step S3.2: Contextual Modulation via FiLM

[0094] In order to reflect the adjusting effect of the geographical background, first extract the hidden state vector of the first position of the geographical context matrix , then generate the modulation parameters: scaling vector and translation vector :

[0095]

[0096] ​​

[0097] The application feature-wise linear modulation (FiLM) generates the final context guide signal :

[0098]

[0099] wherein, denotes element-wise multiplication.

[0100] This affine transformation allows the geographical information to dynamically enhance or suppress specific dimensions of the intended features.

[0101] Step S4: Guided large language model recommendation

[0102] The context guide signal is spliced before the user intent representation, input into the parameter frozen large language model, obtains the user preference vector and calculates the recommendation score with the candidate location, and the candidate location is sorted according to the recommendation score, and the user is recommended with a high score. Specifically:

[0103] The generated context guide signal is regarded as a soft prompt and spliced into the dialogue intent before the input of the large language model is constructed :

[0104]

[0105] The input is input into the parameter frozen large language model , and the hidden state of the last Token is extracted as the user preference vector :

[0106]

[0107] wherein, the subscript represents the hidden state of the last Token extracted;

[0108] The dot product of the user preference vector and the candidate location embedding is taken as the recommendation score, and the candidate location is sorted according to the recommendation score, and the user is recommended with a high score.

[0109] In this embodiment, the dialogue recommendation system constructed based on the application adopts a cross-entropy loss function for optimization:

[0110]

[0111] wherein, embedding for a real target location, embedding for a candidate location embedding set one candidate location embedding in the embedding set.

[0112] In the optimization process, only the parameters of the external fusion module, i.e., the hierarchical geocoder, the context feature modulation module, and the projection network, are updated, while the parameters of the large language model remain unchanged.

[0113] Experiments

[0114] 1. Experimental setup

[0115] Dataset: We mainly conduct experiments on the Cross-WOZ dataset (Zhu et al. 2020) and increase the coordinates by Amap API to parse all the text addresses. In addition, we use the MultiWOZ2.2 dataset (Zang et al. 2020) to verify the generalization.

[0116] Evaluation metrics: To evaluate the recommendation performance, we use three widely used ranking metrics: Recall@k, NDCG@k, and MRR@k (k ∈ {1, 5, 10}). For all ranking metrics, the higher the value, the better the performance.

[0117] Baselines: We select the following representative baselines for comparison to evaluate the effectiveness of the invention: Popularity, BERT (Devlin et al. 2019), SASRec (Kang and McAuley 2018), KBRD (Chen et al. 2019), KGSF (Zhou et al. 2020a), TG-ReDial (Zhou et al. 2020b), UniCRS (Wang et al. 2022), and ZSCRS (He et al. 2023). DeepSeek-V3 model, VRICR (Zhang et al. 2023), and DCRS (Dao et al. 2024) are used to implement dialogue recommendation.

[0118] 2. Overall performance comparison

[0119] The recommendation performance comparison on the CrossWOZ and MultiWOZ datasets is shown in Table 1:

[0120]

[0121] Table 1

[0122] where the symbol "*" represents a significant improvement compared to the best baseline. From the experimental results in Table 1, we can see that the present application, i.e., the GeoCRS model, outperforms all other baselines on both datasets. For the strongest baseline, on the Cross-WOZ dataset, the present application achieves a significant improvement of 20.9% in the evaluation metric Recall@1. This is because traditional sequential and general-purpose models, such as Popularity, BERT and SASRec, lack the understanding of the dialogue context, while knowledge graph-based methods such as KBRD, KGSF, TG-ReDial and VRICR and general-purpose large language model methods such as ZSCRS, UniCRS and DCRS cannot handle the dynamic, heterogeneous and multi-modal characteristics of geographic information. In contrast, the GeoCRS model proposed by the present application benefits from two key innovations: (1) hierarchical geographic coding, which unifies geographic data into a rich representation for large language models, and (2) context feature modulation, which modulates dynamic dialogue intent with geographic context through an asymmetric fusion mechanism, thus achieving true geographic attention reasoning and improving recommendation performance.

[0123] Although the above describes the specific embodiments of the present application in order to facilitate the understanding of the present application by those skilled in the art, it should be clear that the present application is not limited to the scope of the specific embodiments, and that all applications utilizing the concept of the present application are within the scope of protection of the present application, as long as various changes are obvious to those skilled in the art within the spirit and scope of the present application as defined and determined by the appended claims.

Claims

1. A large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation, characterized in that, Includes the following steps: (1) Hierarchical geocoding; The location sequence in the user's interaction history is obtained. The text address, geographic coordinates and numerical attributes of each location in the location sequence are encoded and fused to generate a location representation. All location representations are arranged in rows to obtain a geographic context matrix. (2) User intent modeling; Obtain the current dialogue history text, extract semantic features using a pre-trained language model, and generate a representation of user intent; (3) Context feature modulation; By using the geographic context matrix as a condition variable, an asymmetric fusion mechanism is used to dynamically modulate the user intent representation, generating a contextual guidance signal containing geographic constraints. (4) Guided recommendation based on large language models; The contextual guidance signal is treated as a soft cue and concatenated before the user's intent representation. It is then input into a large language model with frozen parameters to obtain the user preference vector and embed it with the candidate positions to calculate the recommendation score. Based on the recommendation score, the candidate positions are sorted, and the high-scoring positions are recommended to the user.

2. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 1, characterized in that, The text address encoding for each position in the position sequence described in step (1) is as follows: The text address is divided into multiple spatial units according to administrative level or semantic structure. The feature vector of each spatial unit is extracted using a text encoder. A learnable query vector is introduced, the attention weight of each spatial unit is calculated, and the feature vectors are weighted and summed based on the weights to obtain a unified address representation.

3. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 2, characterized in that, The text address encoding for each position in the position sequence described in step (1) is as follows: First, the text address Divided into spatial units ,in, Indicates the index of a position in a position sequence. This represents the text address in the location; Extract the feature vector of each spatial unit using a pre-trained text encoder. : ； in, Indicates the sequence number of the spatial unit. Indicates average pooling. This refers to the RoBERTa pre-trained text encoder; Introduce a learnable query vector Calculate attention weights : ； in, Indicates transpose. This indicates the sequence number of the spatial unit; Final Location address representation For weighted sum: 。 4. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 1, characterized in that, Step (1) encodes the geographic coordinates of each location in the location sequence as follows: The geographic coordinates are latitude and longitude spherical coordinates, and the latitude and longitude coordinates of the location are converted into local planar Cartesian coordinates using a map projection algorithm; Multi-scale sine functions are used to encode local planar Cartesian coordinates to generate position features at different spatial frequencies; By stitching together location features at different scales, we obtain the first... Multi-scale coordinate representation of location .

5. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 4, characterized in that, Step (1) encodes the geographic coordinates of each location in the location sequence as follows: First, the Lambert Conformal Conic (LCC) projection is used to project the first... Longitude of the location and latitude Convert to local plane coordinates : Calculate projection constants and : ； ； in, These are the first and second standard parallels of latitude, respectively. Calculate polar coordinates : ； ； in, For reference longitude; Convert to Cartesian coordinates: ； in, The radius at the reference dimension; Then, for local planar coordinates Perform multi-scale sinusoidal coding, for each scale The encoding formula is: ； ； in, This represents the total number of frequency dimensions at each scale; frequency Based on the maximum radius definition: ； Ultimately, multi-scale sinusoidal coding of all scales will be achieved. By concatenating the coordinates, we obtain the coordinate representation. .

6. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 1, characterized in that, In step (1), the numerical attribute of each position in the position sequence is encoded as follows: First, construct the input embedding sequence. : will the first The values ​​of each numeric attribute are mapped to a numeric embedding vector. , The total number of numerical attributes, and the learnable type embedding vectors are superimposed on each of them. To distinguish attribute categories, a classification tag embedding for aggregating global information is introduced at the beginning of the sequence. and its type embedding The calculation formula is as follows: ； Next, the constructed input embedding sequence The input is a Transformer encoder to capture deep dependencies between attributes, and the hidden states corresponding to the CLS markers in the output layer are extracted as a unified feature representation of the numerical attributes at that location. : 。 7. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 1, characterized in that, The location representations generated in step (1) are merged and arranged in rows to obtain the geographic context matrix: The three representations are concatenated and fused using a multilayer perceptron (MLP) to generate a location representation. : ； Arrange the location representations of all locations in the user interaction history by row to obtain the geographic context matrix. .

8. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 1, characterized in that, Step (3) involves using the geographic context matrix as a condition variable and employing an asymmetric fusion mechanism to dynamically modulate the user intent representation, generating a contextual guidance signal containing geographic constraints: Step 3.1) Content alignment; By utilizing the cross-attention mechanism, we can identify the expression of user intent. The most relevant geographical history is attention. , to obtain the alignment representation : ； ； Step 3.2), Context Modulation; First, extract the geographic context matrix. The hidden state vector at the first position is obtained Then through the projection network Generate modulation parameters: scaling vector Translation vector : ； ； The final context-guided signal is generated by applying characteristic linear modulation. : ； in, This indicates element-wise multiplication.

9. The large language model dialogue recommendation method based on multimodal geographic information fusion and context modulation according to claim 1, characterized in that, Step (3) involves treating the contextual guidance signal as a soft cue and concatenating it before the user's intent representation. This cue is then input into a large language model with frozen parameters. The user preference vector is obtained and embedded with candidate positions to calculate a recommendation score. Based on the recommendation score, the candidate positions are sorted, and those with higher scores are recommended to the user as follows: Generate context guide signal Treat it as a soft cue and splice it into the dialogue intent. Previously, the input for building large language models : ； Input Large language models with frozen input parameters Extract the hidden state of the last token as a user preference vector. : ； Among them, subscript This indicates the hidden state of retrieving the last token; Calculate user preference vector embedding with candidate positions The dot product is used as the recommendation score. Based on the recommendation score, the candidate positions are sorted, and the positions with higher scores are recommended to the user.

Images