Generative dialogue recommendation system and method based on large language model
By using a generative dialogue recommendation system based on a large language model, and leveraging user memory behavior towers, current context towers, and user profile towers for information extraction and fusion, this system solves the problem of generating new questions and capturing intent in existing question recommendation systems, and achieves high-quality question recommendation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-25
- Publication Date
- 2026-03-13
AI Technical Summary
Existing question recommendation systems are unable to generate new questions, struggle to capture users' deep-seated and dynamically evolving interests, and lack transparent and easily understandable reasons for recommendation results, leading to information cocoons and suppressed exploratory discovery.
A generative dialogue recommendation system based on a large language model is adopted. By jointly training the user memory behavior tower, the current context tower, and the user profile tower, and combining feature fusion and decoder, the top-K most likely question recommendation sequences are generated. The Transformer encoder and multi-layer deep neural network are used for information extraction and fusion.
It achieves user intent mining and semantic matching, and generates questions with high relevance, fluency and usefulness. It is suitable for scenarios such as Q&A communities, knowledge platforms, intelligent customer service and online education, with an average score of over 4.5.
Smart Images
Figure CN121658600A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information retrieval technology, and in particular to a generative dialogue recommendation system and method based on a large language model. Background Technology
[0002] Traditional question recommendation systems mainly rely on collaborative filtering (such as user-question interaction matrices) or content matching (such as keyword and topic models), which have obvious drawbacks:
[0003] (1) Inventory limitation: Only issues in the existing issue library can be recommended; new or unseen issues cannot be generated.
[0004] (2) Shallow association: It relies on co-occurrence statistics or shallow semantic matching, making it difficult to capture users' deep and dynamically evolving interests and intentions.
[0005] (3) Filter bubbles: They tend to over-recommend similar questions that users have encountered in the past, leading to information cocoons and inhibiting exploratory discovery.
[0006] (4) Poor interpretability: The recommendation results lack transparent and easy-to-understand reasons, making it difficult for users to understand "why this issue is recommended".
[0007] Recent research, such as generative recommendation using large language models, has demonstrated the enormous potential of LLM (Large Language Model) in recommender systems, but its main focus has been on product recommendation. How to apply the intent mining and generation capabilities of LLM to the novel field of question recommendation remains an unsolved challenge.
[0008] Therefore, it is necessary to provide a generative dialogue recommendation system and method based on a large language model to perform user intent mining, semantic matching recommendation, and recommendation problem generation. Summary of the Invention
[0009] The purpose of this invention is to provide a generative dialogue recommendation system and method based on a large language model for user intent mining, semantic matching recommendation, and generation of recommendation questions.
[0010] To address the problems existing in the prior art, this invention provides a generative dialogue recommendation system based on a large language model, comprising:
[0011] A three-tower neural network model is formed by jointly training three independent encoders. The three-tower neural network model includes: user memory behavior tower, current context tower, user profile tower, and feature fusion and decoder.
[0012] The training module is configured to build training data, calculate the loss function, and complete the training to obtain a generative dialogue recommendation model.
[0013] The reasoning and recommendation functions are configured based on a generative dialogue recommendation model and input triples from real users, outputting the top-K most likely question recommendation sequences.
[0014] Optionally, in the generative dialogue recommendation system based on the large language model, the input and output of the user memory behavior tower are as follows:
[0015] Input: A sequence of questions recently asked by the user, where the question sequence is H = [h1, h2, ..., hn];
[0016] Structure: Using a Transformer encoder or LSTM network, each historical question is first converted into a vector through a word embedding layer, and then the sequence dependency is captured through a temporal model;
[0017] Output: A fixed-length vector representing the user's short-term interests and intent evolution;
[0018] The input and output of the current context tower are as follows:
[0019] Input: The text content that the user is currently viewing or asking a question;
[0020] Structure: A pre-trained language model is used as the encoder to fully utilize its powerful semantic understanding capabilities;
[0021] Output: A fixed-length context semantic vector representing the user's immediate intent;
[0022] The input and output of the user profile tower are as follows:
[0023] Input: The user's static characteristics, including demographic information, family members, zodiac sign, and MBTI;
[0024] Structure: Multi-layer deep neural networks are used to encode and fuse structured features;
[0025] Output: A fixed-length user profile vector representing the user's long-term stable interests;
[0026] Feature fusion and decoder:
[0027] Fusion: The output vectors of the three towers are concatenated or weighted and summed to form the final joint representation vector;
[0028] Decoding: The problem of generating recommendations token by token by token, by inputting the final joint representation vector as the initial state into a Transformer-based autoregressive decoder.
[0029] Optionally, in the generative dialogue recommendation system based on the large language model, the training data is constructed by constructing triple samples from platform logs. The user's triple sample includes Memory, Profile, and content, where Memory is the user profile, Profile is the user's profile, and content is the current context content; the training target Target_Question is the next question that the user actually asks in this context and history.
[0030] Optionally, in the generative dialogue recommendation system based on the large language model, the loss function is calculated as follows:
[0031] Using standard cross-entropy loss, the loss between the word prediction distribution at each position in the generated sequence and the real label is calculated and summed. The formula is: Loss=-Σ(y_i*log(p_i)), where y_i is the one-hot encoding of the real token and p_i is the probability distribution predicted by the model.
[0032] Optionally, in the generative dialogue recommendation system based on the large language model, the training is completed in the following way:
[0033] Phased training: First, pre-train the context tower and decoder with massive amounts of data, and then introduce the three towers for joint fine-tuning;
[0034] or,
[0035] Joint training: Direct end-to-end joint training is performed, and the parameters of the three towers and the decoder are updated simultaneously through the backpropagation algorithm, so that the model learns to extract and fuse the three types of information.
[0036] This invention also provides a generative dialogue recommendation method based on a large language model, comprising the following steps:
[0037] Three independent encoders are jointly trained to form a three-tower neural network model, which includes: a user memory behavior tower, a current context tower, a user profile tower, and a feature fusion and decoder.
[0038] By constructing training data, calculating the loss function, and completing the training, a generative dialogue recommendation model is obtained.
[0039] Based on a generative dialogue recommendation model and inputting triples from real users, the model outputs the Top-K most likely question recommendation sequences.
[0040] Optionally, in the generative dialogue recommendation method based on the large language model,
[0041] The input and output of the user memory behavior tower are as follows:
[0042] Input: A sequence of questions recently asked by the user, where the question sequence is H = [h1, h2, ..., hn];
[0043] Structure: Using a Transformer encoder or LSTM network, each historical question is first converted into a vector through a word embedding layer, and then the sequence dependency is captured through a temporal model;
[0044] Output: A fixed-length vector representing the user's short-term interests and intent evolution;
[0045] The input and output of the current context tower are as follows:
[0046] Input: The text content that the user is currently viewing or asking a question;
[0047] Structure: A pre-trained language model is used as the encoder to fully utilize its powerful semantic understanding capabilities;
[0048] Output: A fixed-length context semantic vector representing the user's immediate intent;
[0049] The input and output of the user profile tower are as follows:
[0050] Input: The user's static characteristics, including demographic information, family members, zodiac sign, and MBTI;
[0051] Structure: Multi-layer deep neural networks are used to encode and fuse structured features;
[0052] Output: A fixed-length user profile vector representing the user's long-term stable interests;
[0053] Feature fusion and decoder:
[0054] Fusion: The output vectors of the three towers are concatenated or weighted and summed to form the final joint representation vector;
[0055] Decoding: The problem of generating recommendations token by token by token, by inputting the final joint representation vector as the initial state into a Transformer-based autoregressive decoder.
[0056] Optionally, in the generative dialogue recommendation method based on the large language model, the training data is constructed as follows: triple samples are constructed from platform logs. The user's triple sample includes Memory, Profile, and content, where Memory is the user profile, Profile is the user's profile, and content is the current context content; the training target Target_Question is the next question that the user actually asks in this context and history.
[0057] Optionally, in the generative dialogue recommendation method based on the large language model, the loss function is calculated as follows:
[0058] Using standard cross-entropy loss, the loss between the word prediction distribution at each position in the generated sequence and the real label is calculated and summed. The formula is: Loss=-Σ(y_i*log(p_i)), where y_i is the one-hot encoding of the real token and p_i is the probability distribution predicted by the model.
[0059] Optionally, in the generative dialogue recommendation method based on the large language model, the training is completed in the following way:
[0060] Phased training: First, pre-train the context tower and decoder with massive amounts of data, and then introduce the three towers for joint fine-tuning;
[0061] or,
[0062] Joint training: Direct end-to-end joint training is performed, and the parameters of the three towers and the decoder are updated simultaneously through the backpropagation algorithm, so that the model learns to extract and fuse the three types of information.
[0063] Compared with the prior art, the present invention has the following advantages:
[0064] (1) This invention can perform user intent mining, semantic matching recommendation, and generation of recommendation questions.
[0065] (2) This invention has a wide range of applications and is suitable for scenarios that require problem discovery and recommendation, such as Q&A communities, knowledge platforms, intelligent customer service and online education.
[0066] (3) The questions generated by this invention scored an average of over 4.5 points (out of 5) in terms of relevance, fluency and usefulness, further demonstrating its great potential in practical applications. Attached Figure Description
[0067] Figure 1 This is a module diagram of the generative dialogue recommendation model provided in an embodiment of the present invention. Detailed Implementation
[0068] The specific embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. The advantages and features of the present invention will become clearer from the following description. It should be noted that the drawings are all in a very simplified form and use non-precise proportions, and are only used to facilitate and clarify the illustration of the embodiments of the present invention.
[0069] In the description of this application, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0070] In the following, if the methods described herein include a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the steps described may be omitted and / or some other steps not described herein may be added to the method.
[0071] Traditional question recommendation systems primarily rely on collaborative filtering (such as user-question interaction matrices) or content matching (such as keyword and topic models), methods with significant limitations. Furthermore, recent research, such as generative recommendation using large language models, has demonstrated the enormous potential of LLM (Large Language Model) in recommendation systems, but its main focus has been on product recommendation. How to apply the intent mining and generation capabilities of LLM to the novel field of question recommendation remains an unsolved challenge.
[0072] To address the problems existing in the prior art, this invention provides a generative dialogue recommendation system based on a large language model, such as... Figure 1 As shown, the system includes:
[0073] (1) Jointly train a three-tower neural network model formed by three independent encoders. The three-tower neural network model includes: user memory behavior tower, current context tower, user profile tower, and feature fusion and decoder.
[0074] Specifically, 1) the input and output of the User Memory Tower are as follows:
[0075] Input: A sequence of questions (short-term memory, long-term memory, topics) that the user has recently interacted with, where the question sequence is H = [h1, h2, ..., hn];
[0076] Structure: The Transformer encoder or LSTM network is used to first convert each historical question into a vector through a word embedding layer, and then a temporal model is used to capture the sequence dependencies; where LSTM, Long Short-Term Memory, is a type of recurrent neural network.
[0077] Output: A fixed-length vector V_memory representing the user's short-term interest and intent evolution;
[0078] 2) The input and output of the Current ContextTower are as follows:
[0079] Input: The text content C that the user is currently browsing or asking (such as the current page description or the question just entered);
[0080] Structure: A pre-trained language model (Qwen or Llama) is used as the encoder to fully utilize its powerful semantic understanding capabilities;
[0081] Output: A fixed-length context semantic vector V_context, representing the user's immediate intent;
[0082] 3) The input and output of User Profile Tower are as follows:
[0083] Input: The user's static characteristics, which include demographic information (such as age and region), family members, zodiac sign, MBTI, etc. Among them, the letters in MBTI (Myers-Briggs Type Indicator) represent preferences in four dimensions, and each letter corresponds to a personality trait.
[0084] Structure: Multi-layer deep neural networks (DNNs) are used to encode and fuse structured features;
[0085] Output: A fixed-length user profile vector V_profile, representing the user's long-term stable interests;
[0086] 4) Feature Fusion and Decoder:
[0087] Fusion: The output vectors of the three towers are concatenated or weighted and summed (attention-based fusion) to form the final joint representation vector V_final = [V_history; V_context; V_profile];
[0088] Decoding: The final joint representation vector V_final is used as the initial state input into a Transformer-based autoregressive decoder to generate recommendations Q_recommended for each token.
[0089] (2) The training module is configured to build training data, calculate the loss function, and complete the training to obtain a generative dialogue recommendation model. Typically, the training of the model is a supervised learning process with the goal of minimizing the difference between the generated question and the target question (the real question).
[0090] Furthermore, the training data is constructed by building triplet samples from platform logs. The user's triplet sample includes Memory, Profile, and content. Memory is the user profile, Profile is the current context content, and content is the training target Target_Question is the next question that the user actually asks in this context and history.
[0091] The loss function is calculated as follows:
[0092] The standard cross-entropy loss is used to calculate the loss between the word prediction distribution and the real label at each position in the generated sequence, and the summation is given by the formula: Loss=-Σ(y_i*log(p_i)), where y_i is the one-hot encoding of the real token and p_i is the probability distribution predicted by the model.
[0093] The training can be completed in the following ways:
[0094] Phased training (optional): First, pre-train the context tower and decoder using massive amounts of data, then introduce the three towers for joint fine-tuning. Fine-tuning in deep learning refers to the process of further training a pre-trained model to adapt it to a specific task or dataset. Fine-tuning enables the model to perform better on a specific task by training it on a smaller new dataset based on the pre-trained model, without having to train the entire model from scratch. This approach not only saves time and computational resources but also effectively improves the model's performance on specific tasks.
[0095] Alternatively, joint training: Direct end-to-end joint training is performed, and the parameters of the three towers and the decoder are updated simultaneously through the backpropagation algorithm, so that the model learns to extract and fuse the three types of information.
[0096] Reasoning and recommendation are configured based on a generative dialogue recommendation model and a triple (Memory, Context, Profile) input from a real user. The model generates the Top-K most likely question recommendation sequences autoregressively through the aforementioned forward propagation process. Ideally, diverse candidate questions can be generated through beam search, and then presented to the user after combining them with business rules (such as filtering sensitive words).
[0097] Preferably, the generative dialogue recommendation model can achieve online or offline recommendations.
[0098] This invention also provides a generative dialogue recommendation method based on a large language model, comprising the following steps:
[0099] S1: Jointly train three independent encoders to form a three-tower neural network model. The three-tower neural network model includes: user memory behavior tower, current context tower, user profile tower, and feature fusion and decoder.
[0100] Specifically, 1) the input and output of the User Memory Tower are as follows:
[0101] Input: A sequence of questions (short-term memory, long-term memory, topics) that the user has recently interacted with, where the question sequence is H = [h1, h2, ..., hn];
[0102] Structure: The Transformer encoder or LSTM network is used to first convert each historical question into a vector through a word embedding layer, and then a temporal model is used to capture the sequence dependencies; where LSTM, Long Short-Term Memory, is a type of recurrent neural network.
[0103] Output: A fixed-length vector V_memory representing the user's short-term interest and intent evolution;
[0104] 2) The input and output of the Current ContextTower are as follows:
[0105] Input: The text content C that the user is currently browsing or asking (such as the current page description or the question just entered);
[0106] Structure: A pre-trained language model (Qwen or Llama) is used as the encoder to fully utilize its powerful semantic understanding capabilities;
[0107] Output: A fixed-length context semantic vector V_context, representing the user's immediate intent;
[0108] 3) The input and output of User Profile Tower are as follows:
[0109] Input: The user's static characteristics, which include demographic information (such as age and region), family members, zodiac sign, MBTI, etc. Among them, the letters in MBTI (Myers-Briggs Type Indicator) represent preferences in four dimensions, and each letter corresponds to a personality trait.
[0110] Structure: Multi-layer deep neural networks (DNNs) are used to encode and fuse structured features;
[0111] Output: A fixed-length user profile vector V_profile, representing the user's long-term stable interests;
[0112] 4) Feature Fusion and Decoder:
[0113] Fusion: The output vectors of the three towers are concatenated or weighted and summed (attention-based fusion) to form the final joint representation vector V_final = [V_history; V_context; V_profile];
[0114] Decoding: The final joint representation vector V_final is used as the initial state input into a Transformer-based autoregressive decoder to generate recommendations Q_recommended for each token.
[0115] S2: Construct training data, calculate the loss function, and complete training to obtain a generative dialogue recommendation model; typically, model training is a supervised learning process with the goal of minimizing the difference between the generated question and the target question (the real question).
[0116] Furthermore, the training data is constructed by building triplet samples from platform logs. The user's triplet sample includes Memory, Profile, and content. Memory is the user profile, Profile is the current context content, and content is the training target Target_Question is the next question that the user actually asks in this context and history.
[0117] The loss function is calculated as follows:
[0118] The standard cross-entropy loss is used to calculate the loss between the word prediction distribution and the real label at each position in the generated sequence, and the summation is given by the formula: Loss=-Σ(y_i*log(p_i)), where y_i is the one-hot encoding of the real token and p_i is the probability distribution predicted by the model.
[0119] The training can be completed in the following ways:
[0120] Phased training (optional): First, pre-train the context tower and decoder using massive amounts of data, then introduce the three towers for joint fine-tuning. Fine-tuning in deep learning refers to the process of further training a pre-trained model to adapt it to a specific task or dataset. Fine-tuning enables the model to perform better on a specific task by training it on a smaller new dataset based on the pre-trained model, without having to train the entire model from scratch. This approach not only saves time and computational resources but also effectively improves the model's performance on specific tasks.
[0121] Alternatively, joint training: Direct end-to-end joint training is performed, and the parameters of the three towers and the decoder are updated simultaneously through the backpropagation algorithm, so that the model learns to extract and fuse the three types of information.
[0122] S3: Based on the generative dialogue recommendation model and the triples (Memory, Context, Profile) of the input real users, the model generates the Top-K most likely question recommendation sequences autoregressively through the above forward propagation process; preferably, diverse candidate questions can be generated through beam search, and then presented to the user after combining business rules (such as filtering sensitive words).
[0123] Preferably, the generative dialogue recommendation model can achieve online or offline recommendations.
[0124] In summary, compared with the prior art, the present invention has the following advantages:
[0125] (1) This invention can perform user intent mining, semantic matching recommendation, and generation of recommendation questions.
[0126] (2) This invention has a wide range of applications and is suitable for scenarios that require problem discovery and recommendation, such as Q&A communities, knowledge platforms, intelligent customer service and online education.
[0127] (3) The questions generated by this invention scored an average of over 4.5 points (out of 5) in terms of relevance, fluency and usefulness, further demonstrating its great potential in practical applications.
[0128] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A generative dialogue recommendation system based on a large language model, characterized in that, include: A three-tower neural network model is formed by jointly training three independent encoders. The three-tower neural network model includes: user memory behavior tower, current context tower, user profile tower, and feature fusion and decoder. The training module is configured to build training data, calculate the loss function, and complete the training to obtain a generative dialogue recommendation model. The reasoning and recommendation functions are configured based on a generative dialogue recommendation model and input triples from real users, outputting the top-K most likely question recommendation sequences.
2. The generative dialogue recommendation system based on a large language model as described in claim 1, characterized in that, The input and output of the user memory behavior tower are as follows: Input: A sequence of questions recently asked by the user, where the question sequence is H = [h1, h2, ..., hn]; Structure: Using a Transformer encoder or LSTM network, each historical question is first converted into a vector through a word embedding layer, and then the sequence dependency is captured through a temporal model; Output: A fixed-length vector representing the user's short-term interests and intent evolution; The input and output of the current context tower are as follows: Input: The text content that the user is currently viewing or asking a question; Structure: A pre-trained language model is used as the encoder to fully utilize its powerful semantic understanding capabilities; Output: A fixed-length context semantic vector representing the user's immediate intent; The input and output of the user profile tower are as follows: Input: The user's static characteristics, including demographic information, family members, zodiac sign, and MBTI; Structure: Multi-layer deep neural networks are used to encode and fuse structured features; Output: A fixed-length user profile vector representing the user's long-term stable interests; Feature fusion and decoder: Fusion: The output vectors of the three towers are concatenated or weighted and summed to form the final joint representation vector; Decoding: The problem of generating recommendations token by token by token, by inputting the final joint representation vector as the initial state into a Transformer-based autoregressive decoder.
3. The generative dialogue recommendation system based on a large language model as described in claim 2, characterized in that, The training data is constructed by building triple samples from platform logs. The user's triple sample includes Memory, Profile, and content. Memory is the user profile, Profile is the current context content, and content is the training target Target_Question is the next question that the user actually asks in this context and history.
4. The generative dialogue recommendation system based on a large language model as described in claim 3, characterized in that, The loss function is calculated as follows: Using standard cross-entropy loss, the loss between the word prediction distribution at each position in the generated sequence and the real label is calculated and summed. The formula is: Loss=-Σ(y_i*log(p_i)), where y_i is the one-hot encoding of the real token and p_i is the probability distribution predicted by the model.
5. The generative dialogue recommendation system based on a large language model as described in claim 4, characterized in that, The training can be completed in the following ways: Phased training: First, pre-train the context tower and decoder with massive amounts of data, and then introduce the three towers for joint fine-tuning; or, Joint training: Direct end-to-end joint training is performed, and the parameters of the three towers and the decoder are updated simultaneously through the backpropagation algorithm, so that the model learns to extract and fuse the three types of information.
6. A generative dialogue recommendation method based on a large language model, characterized in that, Includes the following steps: Three independent encoders are jointly trained to form a three-tower neural network model, which includes: a user memory behavior tower, a current context tower, a user profile tower, and a feature fusion and decoder. By constructing training data, calculating the loss function, and completing the training, a generative dialogue recommendation model is obtained. Based on a generative dialogue recommendation model and inputting triples from real users, the model outputs the Top-K most likely question recommendation sequences.
7. The generative dialogue recommendation method based on a large language model as described in claim 6, characterized in that, The input and output of the user memory behavior tower are as follows: Input: A sequence of questions recently asked by the user, where the question sequence is H = [h1, h2, ..., hn]; Structure: Using a Transformer encoder or LSTM network, each historical question is first converted into a vector through a word embedding layer, and then the sequence dependency is captured through a temporal model; Output: A fixed-length vector representing the user's short-term interests and intent evolution; The input and output of the current context tower are as follows: Input: The text content that the user is currently viewing or asking a question; Structure: A pre-trained language model is used as the encoder to fully utilize its powerful semantic understanding capabilities; Output: A fixed-length context semantic vector representing the user's immediate intent; The input and output of the user profile tower are as follows: Input: The user's static characteristics, including demographic information, family members, zodiac sign, and MBTI; Structure: Multi-layer deep neural networks are used to encode and fuse structured features; Output: A fixed-length user profile vector representing the user's long-term stable interests; Feature fusion and decoder: Fusion: The output vectors of the three towers are concatenated or weighted and summed to form the final joint representation vector; Decoding: The problem of generating recommendations token by token by token, by inputting the final joint representation vector as the initial state into a Transformer-based autoregressive decoder.
8. The generative dialogue recommendation method based on a large language model as described in claim 7, characterized in that, The training data is constructed as follows: triplet samples are constructed from platform logs. The user's triplet sample includes Memory, Profile, and content. Memory is the user profile, Profile is the current context content, and content is the training target Target_Question is the next question that the user actually asks in this context and history.
9. The generative dialogue recommendation method based on a large language model as described in claim 8, characterized in that, The loss function is calculated as follows: Using standard cross-entropy loss, the loss between the word prediction distribution at each position in the generated sequence and the real label is calculated and summed. The formula is: Loss=-Σ(y_i*log(p_i)), where y_i is the one-hot encoding of the real token and p_i is the probability distribution predicted by the model.
10. The generative dialogue recommendation method based on a large language model as described in claim 9, characterized in that, The training can be completed in the following ways: Phased training: First, pre-train the context tower and decoder with massive amounts of data, and then introduce the three towers for joint fine-tuning; or, Joint training: Direct end-to-end joint training is performed, and the parameters of the three towers and the decoder are updated simultaneously through the backpropagation algorithm, so that the model learns to extract and fuse the three types of information.