A method and system for multimodal search of chat logs based on clip and llm

Abstract

The invention belongs to the technical field of information retrieval, and provides a chat record multi-modal search method and system based on CLIP and LLM, and the LLM is used for carrying out intention analysis and semantic enhancement on information in user query and chat records. Uniformly encoding texts, images and voice information in user query and chat records into the same semantic space through CLIP multi-mode encoding; the multi-modal feature vectors are stored in a vector database, and dynamic updating and efficient retrieval are achieved based on a hierarchical index structure; dynamic weighted sorting of semantic correlation, time decay weight and user preference is carried out on retrieval results; and displaying the multi-modal search result containing the text, the image, the image-text mixture or the voice abstract to the user. Through CLIP cross-modal semantic mapping and LLM intention completion, deep semantic association can be captured, semantic understanding can be optimized, natural language query analysis can be supported, multi-modal information acquisition efficiency can be improved, and user interaction experience can be optimized.

Description

(19) State Intellectual Property Office (12) Invention Patent Application (10) Application Publication Number (43) Application Publication Date (21) Application Number 202511479178.3 (22) Application Date 2025.10.16 (71) Applicant CITIC International Telecommunications (Information Technology) Limited Address 20 / F, Lincoln Tower, Taikoo Place, 979 King's Road, Quarry Bay, Hong Kong Applicant China Enterprise Network Communication Technology Co., Ltd. (72) Inventor Li Chaoqun (74) Patent Agency Wuxi Suying Patent Agency Co., Ltd. 32787 Patent Attorney Tian Deyu (51) Int.Cl. G06F 16 / 41 (2019.01) G06F 16 / 432 (2019.01) G06F 16 / 435 (2019.01) G06F 16 / 438 (2019.01) G06N 3 / 0455 (2023.01) (54) Invention Title: A Multimodal Search Method and System for Chat Logs Based on CLIP and LLM (57) Abstract: This invention belongs to the field of information retrieval technology and provides a multimodal search method and system for chat logs based on CLIP and LLM. It utilizes LLM to perform intent parsing and semantic enhancement on information in user queries and chat logs. Through CLIP multimodal encoding, text, images, and voice information in user queries and chat logs are uniformly encoded into the same semantic space. Multimodal feature vectors are stored in a vector database and dynamically updated and efficiently retrieved based on a hierarchical index structure. The retrieval results are dynamically weighted and sorted based on semantic relevance, time decay weight, and user preferences. The system displays multimodal search results containing text, images, mixed text and images, or voice summaries to the user. This invention, through CLIP cross-modal semantic mapping and LLM intent completion, can capture deep semantic relationships, optimize semantic understanding, support natural language query parsing, improve the efficiency of multimodal information acquisition, and optimize the user interaction experience. Claims (2 pages), Description (9 pages), Drawings (2 pages), CN 121434418 A, 2026.01.30, CN 1 21 43 44 18 A. 1. A multimodal search method for chat logs based on CLIP and LLM, characterized in that the method includes: using LLM to perform intent parsing and semantic enhancement on information in user queries and chat logs, and then using CLIP multimodal encoding to uniformly encode text, image, and voice information in user queries and chat logs into the same semantic space; storing multimodal feature vectors in a vector database, and achieving dynamic updates and efficient retrieval based on a hierarchical index structure; dynamically weighting the retrieval results based on semantic relevance, time decay weight, and user preference using a hybrid ranking strategy; and displaying multimodal search results to the user, including text, images, mixed text and image data, or voice summaries.2. The chat history multimodal search method based on CLIP and LLM according to claim 1, characterized in that, in the process of using LLM to perform intent parsing and semantic enhancement on user queries and information in chat history, LLM is used to complete the intent of input information and generate an enhanced vector containing time, entity, sentiment, and topic. 3. The chat history multimodal search method based on CLIP and LLM according to claim 1, characterized in that the specific steps of the CLIP multimodal encoding include: encoding text information into semantic vectors using the CLIP Transformer architecture; extracting visual feature vectors from image information using the CLIP visual encoder; and generating semantic vectors from speech information through ASR conversion or audio feature extraction paths. 4. The chat history multimodal search method based on CLIP and LLM according to claim 1, characterized in that, the dynamic updating and efficient retrieval based on the hierarchical index structure includes: constructing a hierarchical index structure containing a time index and a user / group index; using an incremental learning mechanism to encode new messages in real time and update the index; and using a caching strategy for indexed data, optimizing the caching strategy based on access frequency and the LRU algorithm. 5. The chat history multimodal search method based on CLIP and LLM according to claim 1, characterized in that the scoring formula of the hybrid sorting strategy is: where is the CLIP semantic relevance score, is the time decay weight coefficient, is the user preference coefficient, and , , are normalized weight coefficients. 6. The chat history multimodal search method based on CLIP and LLM according to claim 1, characterized in that the time decay weight coefficient is calculated using an exponential decay function, expressed as: where is the message timestamp, is the current time, and is the weight number. 7. The chat history multimodal search method based on CLIP and LLM according to claim 1, characterized in that the user preference coefficient is generated in the following way: constructing a short-term interest vector and a long-term preference vector based on user historical behavior data; adjusting the weights in combination with explicit feedback signals, expressed as: where is the short-term interest vector, is the long-term preference vector, F is the explicit feedback set, and is the feedback weight coefficient. 8. A chat history multimodal search system based on CLIP and LLM, characterized in that the system comprises: an LLM processing module, used for intent parsing and semantic enhancement of information in user queries and chat history, transforming unstructured multimodal information into structured semantics containing core elements; and a CLIP multimodal encoding module, used for uniformly encoding text, image, and voice information in user queries and chat history into the same semantic space, and performing intent parsing and semantic enhancement based on the large model LLM.The system comprises: a storage module, including a vector database and a hierarchical index structure, for storing multimodal feature vectors and supporting dynamic updates and efficient retrieval; a hybrid ranking strategy module, which dynamically weights and fuses semantic relevance, time decay weights, and user preferences to generate a comprehensive ranking score; and a user interaction module, which performs natural language query parsing and structured information extraction on user input, and displays multimodal search results and speech summary information. 9. The multimodal chat history search system based on CLIP and LLM according to claim 8, characterized in that the CLIP multimodal encoding module includes: a text encoding submodule, based on the CLIP Transformer architecture, for mapping text information from user queries and chat history into high-dimensional semantic vectors; an image encoding submodule, employing a CLIP visual encoder to extract features from visual content and generate semantic vectors aligned with the text; a speech encoding submodule, for generating semantic vectors from speech information through an ASR transformation path or an audio feature extraction path; and an LLM submodule, for converting unstructured multimodal information into structured semantics containing core elements. 10. A server, characterized in that it includes at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to perform the chat history multimodal search method based on CLIP and LLM as described in any one of claims 1 to 6. Claims 2 / 2 Page 3 CN 121434418 A A Chat History Multimodal Search Method and System Based on CLIP and LLM Technical Field

[0001] This invention belongs to the field of information retrieval technology, specifically relating to a chat history multimodal search method and system based on CLIP and LLM, applicable to information retrieval scenarios in various instant messaging software, online collaboration platforms, and enterprise-level communication tools. Background Technology

[0002] With the widespread application of instant messaging (IM) software, users generate a large amount of information during chat, including text messages, pictures, emoticons, voice messages, and files. This rich information constitutes complex multimodal data, posing a challenge to users' information retrieval. In existing instant messaging systems, chat history search functions mainly rely on keyword matching technology. Users input keywords, and the system searches the text content of chat logs, returning records containing those keywords. However, this traditional search method has the following limitations: 1. Single-modal search limitation: Existing search functions mainly target text information and cannot effectively handle non-text content such as images, voice messages, and emoticons. 2. Insufficient semantic understanding: Keyword matching technology is based solely on direct character matching.3. Timeliness issues: With chat history constantly being updated, traditional search systems usually need to build indexes periodically, which cannot reflect the latest chat content in real time. This may cause delays for users when searching for the latest messages, failing to meet the high timeliness requirements of instant messaging. 4. Insufficient personalization: Existing search functions usually adopt a unified search strategy and have a single search method, only supporting text search, and cannot be optimized according to users' personal preferences and usage habits.

[0003] Therefore, this invention proposes a multimodal search method and system for instant messaging chat history based on Contrastive Language-Image Pretraining (CLIP) and Large Language Model (LLM). By enhancing information through LLM, the CLIP model maps information from multiple modalities such as text and images to the same semantic space, realizing cross-modal semantic matching. Summary of the Invention

[0004] The purpose of this invention is to overcome the existing defects and provide a multimodal search method and system for chat history based on CLIP and LLM, which can effectively improve the search efficiency and accuracy of chat history.

[0005] To solve the above technical problems, the present invention provides the following technical solution: The first objective of the present invention is to provide a multimodal search method for chat records based on CLIP and LLM, including: using LLM to perform intent parsing and semantic enhancement on information in user queries and chat records, and then using CLIP multimodal encoding to uniformly encode text, image, and voice information in user queries and chat records into the same semantic space; storing multimodal feature vectors in a vector database, and realizing dynamic updates and efficient retrieval based on a hierarchical index structure; performing dynamic weighted sorting of retrieval results based on semantic relevance, time decay weight, and user preference based on a hybrid sorting strategy; and displaying multimodal search results containing text, images, mixed text and image, or voice summaries to the user. Specification 1 / 9 pages 4 CN 121434418 A

[0006] Further, in the process of using LLM to perform intent parsing and semantic enhancement on information in user queries and chat records, LLM is used to perform intent completion on the input information to generate an enhanced vector containing time, entity, sentiment, and topic.

[0007] Further, the specific steps of CLIP multimodal encoding include: encoding text information into semantic vectors using CLIP's Transformer architecture; extracting visual feature vectors from image information using CLIP's visual encoder; and generating semantic vectors from speech information through ASR conversion or audio feature extraction paths.

[0008] Further, the implementation of dynamic updates and efficient retrieval based on a hierarchical index structure includes: constructing a hierarchical index structure containing a time index and a user / group index;An incremental learning mechanism is adopted to encode new messages in real time and update the index; a caching strategy is adopted for indexed data, and the caching strategy is optimized based on access frequency and LRU algorithm.

[0009] Further, the scoring formula of the hybrid sorting strategy is: where is the CLIP semantic relevance score, is the time decay weight coefficient, is the user preference coefficient, and , , and are normalized weight coefficients.

[0010] Further, the time decay weight coefficient is calculated using an exponential decay function, expressed as: where is the message timestamp, and is the current time.

[0011] Further, the user preference coefficient is generated in the following way: a short-term interest vector and a long-term preference vector are constructed based on user historical behavior data; the weights are adjusted by combining explicit feedback signals, expressed as: where is the short-term interest vector, is the long-term preference vector, F is the explicit feedback set, and is the feedback weight coefficient.

[0012] Another objective of the present invention is to provide a multimodal search system for chat logs based on CLIP and LLM, comprising: an LLM processing module for performing intent parsing and semantic enhancement on information in user queries and chat logs, transforming unstructured multimodal information into structured semantics containing core elements; a CLIP multimodal encoding module for uniformly encoding text, image, and voice information in user queries and chat logs into the same semantic space, and performing intent parsing and semantic enhancement based on a large model LLM; a storage module comprising a vector database and a hierarchical index structure for storing multimodal feature vectors and supporting dynamic updates and efficient retrieval; a hybrid ranking strategy module for dynamically weighting and fusing semantic relevance, time decay weights, and user preferences to generate a comprehensive ranking score; and a user interaction module for performing natural language query parsing and structured information extraction on user input, and displaying multimodal search results and voice summary information.

[0013] Further, the CLIP multimodal encoding module includes: a text encoding submodule, based on the CLIP Transformer architecture, used to map the text information of user queries and chat records into high-dimensional semantic vectors; an image encoding submodule, using a CLIP visual encoder to extract features from visual content and generate semantic vectors aligned with the text; and a speech encoding submodule, used to generate semantic vectors from speech information through an ASR conversion path or an audio feature extraction path. Another object of the present invention is to provide a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the CLIP and LLM-based chat record multimodal search method provided by the first object of the present invention.

[0014] Another object of the present invention is to provide a server, including at least one processor, and a processor...The memory is connected to the communication device, and the memory stores instructions that can be executed by the at least one processor. The instructions are executed by the processor to cause the at least one processor to execute the chat history multimodal search method based on CLIP and LLM provided by the first objective of the present invention.

[0015] In combination with the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows: The present invention replaces keyword character matching with CLIP cross-modal semantic mapping and LLM intent completion, captures deep semantic associations, and optimizes and improves the problem of insufficient semantic understanding. The present invention maps text, images, and voice (via ASR or direct encoding) to the same semantic space through the CLIP multimodal encoding module, and combines LLM intent completion to achieve deep semantic alignment, overcome the pain points of "single modality limitation" and "insufficient semantic understanding", and solve the problem that the existing technology uses separate encoding for text, images, and voice (such as BERT for text and Wav2Vec for voice), resulting in weak cross-modal semantic associations and only surface matching.

[0016] This invention uses a vector database to uniformly store multimodal vectors, designs a hierarchical index of "time + user / group", and combines incremental learning to update the index in real time with intelligent caching, breaking down storage barriers and achieving millisecond-level response and real-time retrieval. This solves the problems of existing technologies that use heterogeneous storage of multimodal data (text stored in databases, images stored in file systems), require cross-system retrieval and rely on periodic indexing, resulting in low efficiency and poor timeliness.

[0017] This invention dynamically weights and fuses CLIP semantic similarity, time decay weight (exponential function), and user preferences (short-term LSTM + long-term aggregation + explicit feedback). The weights are adjusted online, taking into account semantic accuracy, timeliness, and personalization, which is superior to traditional fixed strategies. This solves the problem that traditional technologies often use fixed weight sorting (such as only by time or keyword matching degree), lacking semantic and personalized adaptation.

[0018] This invention supports natural language query parsing (extracting structured information such as time and people), provides mixed text and image sorting, image search, and voice summarization (highlighting key segments), improving the efficiency of multimodal information acquisition and optimizing the user interaction experience. This invention overcomes the limitations of existing single-modal search technologies, supporting multimodal content retrieval including text, images, voice, and emoticons. It breaks through the limitation of only being able to process text, and addresses the issues of limited interaction methods (such as text-only search) and fragmented multimodal display in existing technologies.

[0019] The accompanying drawings are provided to further illustrate the invention and constitute a part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 is a flowchart of the chat history multimodal search method based on CLIP and LLM provided in an embodiment of the invention; Figure 2 is a schematic diagram of the method provided in an embodiment of the invention; Figure 3 is a flowchart of user interaction provided in an embodiment of the invention;Figure 4 is a structural block diagram of the chat history multimodal search system based on CLIP and LLM provided in the embodiment of the present invention. Specification 3 / 9 pages 6 CN 121434418 A Detailed Embodiments

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of the present invention and are not intended to limit the present invention.

[0021] Embodiment 1: As shown in Figure 1, this is an embodiment of the chat history multimodal search method based on CLIP and LLM provided by the present invention, which specifically includes the following steps: S1: Use LLM to perform intent parsing and semantic enhancement on the information in the user query and chat history, and then use CLIP multimodal encoding to uniformly encode the text, image, and voice information in the user query and chat history into the same semantic space; S2: Store the multimodal feature vectors in the vector database and realize dynamic updates and efficient retrieval based on the hierarchical index structure; S3: Perform dynamic weighted sorting of the retrieval results based on semantic relevance, time decay weight, and user preference based on the hybrid sorting strategy; S4: Display multimodal search results containing text, images, mixed text and image, or voice summaries to the user.

[0022] Specifically, as shown in Figure 2, this embodiment of the invention uses the CLIP multimodal encoding step, employing CLIP's text encoder (such as Transformer) to map text information in user queries and chat logs into semantic vectors; and using CLIP's visual encoder to extract features from images, emoticons, and video keyframes to generate visual semantic vectors. For speech encoding, it first converts to text using ASR (such as Whisper), then sends it to the CLIP text encoder, or combines audio features (such as VGGish) for cross-modal alignment. Based on CLIP, LLM (such as ChatGLM) is introduced to perform intent parsing on the input, generating enhanced vectors containing information such as time, entities, sentiment, and topics. A vector database is used to store the multimodal feature vectors of all chat logs, supporting efficient similarity retrieval. Multiple strategies are combined for sorting, and the search result with the highest score is returned. LLM is used to perform multi-round semantic parsing on the query, supporting image, natural language, and speech queries, and parsing structured information such as time and people. Multimodal search result display is provided, such as: mixed image and text sorting (related images + contextual text), speech summarization, etc.

[0023] The various steps of the present invention will be described in detail below.

[0024] I. CLIP Multimodal Coding Step: This step mainly consists of text encoding, speech encoding, image encoding, and LLM processing, among which image encoding can encode video. Existing technologies rely on keyword character matching and lack semantic understanding capabilities (such as the inability to recognize the semantic association between "meeting" and "conference"). The present invention uses CLIP...Cross-modal semantic mapping and intent completion of LLM can not only understand the deep semantics of text, but also transform the content of images and speech into associative semantic vectors, significantly improving the semantic accuracy of cross-modal matching.

[0025] 1. LLM processing: The content is simultaneously processed by LLM for intent completion. The core of intent completion through LLM is to transform unstructured multimodal information (speech, text, images) into structured semantics containing core elements such as time, people, and events. This not only enriches the semantic dimension of the original information, but also provides a unified input form for the cross-modal encoding of CLIP model. This processing method allows information from different modalities to be directly correlated in the same semantic space through cosine similarity calculation. For example, when a user inputs the text query "team activities next week", the system can simultaneously match related voice records (such as "meeting next Monday"), text messages (such as "discussing projects on Wednesday") and image posters (such as "dinner on Friday"), significantly improving the accuracy of cross-modal search and the depth of semantic understanding. Instruction Manual Page 4 / 9 7 CN 121434418 A

[0026] 2. Text Encoding: The text encoding process of this invention is based on the CLIP Transformer architecture, used to map text information such as user queries and chat logs into high-dimensional semantic vectors. The specific implementation includes the following steps: Input preprocessing: The original text is standardized (e.g., Unicode normalization, special symbol filtering), and the text is converted into a token sequence using sub-word segmentation (BPE algorithm); Embedding representation: The tokens are mapped into vectors through a learnable word embedding matrix (dimension d×|V|, where d is the hidden layer dimension and |V| is the vocabulary size), and positional encoding (sine / learned) is superimposed to preserve sequence information; Feature extraction: Context modeling is performed using a multi-layer Transformer encoder (e.g., 12 layers), each layer containing multi-head self-attention (MHA) and feedforward network (FFN); Semantic aggregation: Fixed-dimensional text semantic vectors (e.g., 512 dimensions) are generated through [CLS] tagging or mean pooling.

[0027] 3. Image Encoding: The image encoding process of this invention uses a CLIP visual encoder (ViT or ResNet variant), which supports feature extraction from visual content such as images, emoticons, and video keyframes to generate semantic vectors aligned with text. The specific process includes: Image preprocessing: The input image is normalized to a fixed resolution (e.g., 224×224) and the channels are standardized (mean / variance adjustment); Feature extraction: Taking the ViT architecture as an example: The image is divided into a 16×16 Patch sequence, linearly projected and then superimposed with position encoding, and global features are extracted through a Transformer encoder; Output mapping: Visual features are aligned to the text semantic space through a projection layer. The module supports a dynamic attention mechanism to enhance the feature weights of key regions.

[0028] 4. Speech Coding: The speech coding process of this invention provides a dual-path scheme to achieve efficient mapping of speech to semantic vectors: (1) ASR Text Conversion Path: The Whisper model is used for speech recognition (ASR), the audio signal SS is converted into text TT, and then processed by the CLIP text encoder, supporting multilingual transcription and text post-processing (such as punctuation restoration and spoken language standardization).

[0029] (2) Audio Feature Direct Coding Path: Log-Mel spectrogram is extracted, high-level acoustic features are extracted through the VGGish network, and then aligned to the CLIP space through a cross-modal projection layer.

[0030] II. Vector Storage: This step consists of hierarchical vector database storage, hierarchical index structure, dynamic index update module and intelligent caching strategy. Its core is to achieve high-timeliness semantic retrieval through multi-level index structure and incremental learning mechanism, while using session isolation strategy and cache optimization algorithm to reduce cross-context interference and improve response speed. Unlike existing technologies: a vector database is used to uniformly store multimodal feature vectors, breaking down the barriers of heterogeneous storage of multimodal data and achieving unified and efficient cross-modal retrieval; an innovative hierarchical structure of "time index + user / group index" is designed to balance time range retrieval and session independence, improving retrieval accuracy; new messages are processed and the index is updated in real time through an incremental learning mechanism, solving the time lag problem of traditional batch updates; an intelligent caching strategy based on access frequency and LRU algorithm is used to specifically optimize the response speed of multimodal retrieval scenarios. These designs together constitute a unique innovation in storage and retrieval mechanisms.

[0031] 1. Vector Database Storage: A vector database is used to store the multimodal feature vectors of all chat records. The vector database can efficiently process high-dimensional vector data, support near nearest neighbor retrieval, and achieve millisecond-level response. Through a deep learning model, multimodal data such as text, images, and audio in chat records are converted into high-dimensional vectors and uniformly stored in the vector database. Instruction manual, page 5 / 9, CN 121434418 A

[0032] 2. Hierarchical index structure: Time index: Stored hierarchically by message timestamp, supporting time range retrieval. Chat records are hierarchically classified by timestamp, with each layer corresponding to a time interval, making it easy to quickly locate chat records within a specific time range.

[0033] User / group index: Classified by session context to avoid cross-session interference. Chat records are classified and stored according to users or groups to ensure data independence between different sessions and improve retrieval accuracy.

[0034] 3. Dynamic index update module: Adopting an incremental learning mechanism, when a new message is generated, the index is encoded and updated in real time to ensure the timeliness of search results. The specific process is as follows: (1) When a new message arrives, its multimodal feature vector is extracted through a pre-trained model. (2) The new message is updated in real time.The data is inserted into the vector database in real time and the index structure is updated. (3) Through the dynamic index update mechanism, the index in the database is always up-to-date, thereby providing the latest retrieval results.

[0035] 4. Caching strategy: The indexed data is cached, and the high-frequency access content is loaded first to improve the response speed. The specific implementation is as follows: (1) The access frequency is sorted by data, and the high-frequency access chat records and their indexes are loaded into the cache; (2) The cache adopts the LRU (Least Recently Used) algorithm to regularly eliminate low-frequency access data to ensure the effective use of cache space. III. Hybrid sorting strategy: This strategy generates a comprehensive sorting score by dynamically weighting and fusing semantic relevance score, time decay weight and user preference coefficient. Its core formula is defined as: Where, is the normalized weight coefficient, which is dynamically adjusted through online learning. Through CLIP-driven cross-modal semantic alignment, dynamic weight adjustment through online learning, refined time decay design, dynamically adapted user preference modeling, and closed-loop feedback mechanism, it differs significantly from traditional solutions such as discrete modality encoding, fixed weight strategies, and complex interest genome modeling, achieving innovation and improvement in semantic matching accuracy, scenario adaptability, and personalized response speed.

[0036] 1. Semantic relevance scoring: Semantic relevance scoring is one of the core components of the hybrid ranking strategy. This scoring mechanism is based on the CLIP model and measures the relevance of content by calculating the similarity between the query vector and the chat record vector. This method can deeply understand the semantic relationship between the query and the content, ensuring that the search results are highly matched with the user's needs at the semantic level. In this way, the system can prioritize recommending content that is semantically closest to the user's query, thereby improving the accuracy and relevance of the search results. Compared with traditional solutions that use discrete encoding such as BERT, Wav2Vec, and visual models and then map them to the shared space through cross-modal adversarial networks, this method more directly captures the deep semantic associations of cross-modal content and reduces information loss in modality conversion.

[0037] 2. Time Weight: Time weight is another important scoring dimension in the hybrid ranking strategy. This mechanism weights content according to the message's writing time, ensuring that more recent content has a higher weight. This design can effectively improve the timeliness of search results, enabling users to quickly obtain the latest information. In scenarios where information is updated rapidly, such as news and social media, the introduction of time weight can ensure that users do not miss important updates, thereby enhancing the practicality and attractiveness of the search system.

[0038] Based on the interval between the message timestamp and the current time, an exponential decay function is constructed: Specification 6 / 9 page 9 CN 121434418 A 3. User Preference: User preference is an important manifestation of personalization in the hybrid ranking strategy. This mechanism is based on the user's historical click behavior.The system learns the user's personalized weights through machine learning algorithms based on the user's interaction data. The system can dynamically adjust the ranking of search results according to the user's interests and preferences, thereby providing a customized search experience for each user. This personalized strategy has stronger dynamism, flexibility and implementation optimization capabilities, and can more accurately capture changes in user interests and improve search accuracy.

[0039] The model distinguishes between short-term interests and long-term preferences by analyzing the user's historical behavior data: (1) Short-term interests: Based on the user's most recent k interaction behaviors, the model captures the user's current instant interest points. Let be the set of content of the user's most recent k interactions. The model encodes these interaction behaviors through an LSTM network to obtain a short-term interest vector. (2) Long-term preferences: Based on the user's historical behavior data over the past T days, the model mines the user's stable preference patterns. Let be the set of content of the user's most recent T interactions. The model obtains a long-term preference vector by aggregating this content.

[0040] The model also combines the user's explicit feedback (such as "irrelevant" marking) for real-time correction. Let F be the set of explicit user feedback. The model adjusts the weights through the feedback signals: ; where is the weight coefficient of the feedback signal, used to balance the influence of explicit feedback on the user preference vector. For example, when a user marks the recommended content as "irrelevant", the model will immediately adjust the weight of related topics and reduce their priority in subsequent recommendations. This dynamic adjustment mechanism enables the model to capture the dynamic changes of user interests more flexibly. Compared with the method of constructing user interest profiles through interest genomes and models, it has stronger adaptability and flexibility.

[0041] IV. User Interaction: The user interaction process introduces LLM and multimodal search technology to realize functions such as natural language query parsing, mixed image and text sorting, image search, and voice summarization. These functions not only improve the user experience, but also make the search process more efficient and convenient, and can meet the diverse needs of users in different scenarios. Compared with traditional solutions, the introduction of LLM technology in the search process can enhance the system's understanding of the retrieved information. At the same time, the model realizes the image search function through CLIP encoding. The user interaction module provides rich multimodal search result display methods to meet the needs of different users. The specific steps are as follows: 1. User submits query: The user inputs the query content through natural language text, image or voice. The system supports multimodal input, including text description, uploaded image or voice command. If it is a voice query, the system will first use speech recognition technology to convert it into text (speech summarization function) for subsequent processing.

[0042] 2. Query parsing and time detection: The system parses the user query and checks whether it contains time-related information (such as date, time range). If the query contains time, the time range is extracted; otherwise, it directly proceeds to keyword processing. At the same time, it proceeds to...A global time-domain search is performed to ensure that time-related queries cover all possible time dimensions.

[0043] 3. Multimodal coding and LLM enhanced understanding: For text queries, LLM is combined for deep semantic analysis to enhance the understanding of query intent (such as natural language query parsing). The CLIP text encoder is then used to convert the query text into a vector representation to enhance the understanding of query intent (such as natural language query parsing). For image queries (image search), LLM is combined for contextual understanding to achieve intelligent interpretation of image content. The CLIP visual encoder is then used to convert the uploaded image into a vector representation. This step ensures that the system can handle mixed text and image queries and provides a unified vector representation for subsequent retrieval.

[0044] 4. Vector database retrieval: The encoded query vector (text or image vector) is input into the vector database specification page 7 / 9, 10 CN 121434418 A, for similarity retrieval. The database stores pre-encoded vectors of documents, images, or other multimedia content. The retrieval process will return a set of preliminary matching candidate results.

[0045] 5. Apply time decay weight: If the query contains a time range, the system applies time decay weight to the search results. This means that newer content will receive higher weight, while older content will receive lower weight, to ensure that the search results match the time relevance (a refinement of global time-domain search).

[0046] 6. Calculate similarity score: Based on the query vector and candidate result vector, calculate the similarity score for each result. The score reflects the semantic matching degree between the content and the query.

[0047] 7. Load personalized preferences: The system loads the user's personalized preferences (such as historical search behavior, interest tags, etc.) and adjusts the similarity score. This step makes the search results more in line with the individual needs of the user and improves the personalized experience.

[0048] 8. Generate final ranking: Combining the similarity score, time decay weight, and personalized preferences, all candidate results are comprehensively ranked. The ranking algorithm ensures that the mixed text and image results (such as mixed text and image ranking) can be optimized according to relevance, timeliness, and user preferences.

[0049] 9. Return topk results: The system returns the ranked topk results to the user. The results may be displayed in various forms, such as lists, grids, or summaries, to meet the needs of different users (such as multimodal search results display).

[0050] This user interaction module constructs an intelligent and efficient search system by introducing LLM and multimodal technology. Its core lies in utilizing the powerful natural language understanding capabilities of LLM to accurately parse user queries (such as "find the product introduction mentioned by Zhang San last week") and automatically extract key structured information such as time and people, thereby deeply understanding the user's intent. The system supports multimodal search, and can not only sort and display results in a mixed text and image format, but also achieve "image search" through CLIP encoding.The module provides a "image" function. In addition, for voice content, the module provides a voice summarization function, which uses LLM analysis to convert to text and highlight relevant segments, enabling users to quickly locate key information. Through a series of functions such as natural language parsing, image and text mixed sorting, image search and voice summarization, the system significantly improves the accuracy, richness and efficiency of search, bringing users a more intuitive and convenient cross-modal information retrieval experience.

[0051] Example 2: As shown in Figure 4, this embodiment of the invention provides a chat history multimodal search system based on CLIP and LLM, including: an LLM processing module, used to perform intent parsing and semantic enhancement on information in user queries and chat history, transforming unstructured multimodal information into structured semantics containing core elements; a CLIP multimodal encoding module, used to uniformly encode text, image and voice information in user queries and chat history into the same semantic space, and perform intent parsing and semantic enhancement based on the large model LLM; a storage module, including a vector database and hierarchical index structure, used to store multimodal feature vectors and support dynamic updates and efficient retrieval; The hybrid ranking strategy module generates a comprehensive ranking score based on the dynamic weighted fusion of semantic relevance, time decay weight, and user preference; the user interaction module is used to perform natural language query parsing and structured information extraction on user input, and to display multimodal search results and speech summary information.

[0052] Preferably, the CLIP multimodal encoding module in this embodiment includes: a text encoding submodule, based on the CLIP Transformer architecture, used to map the text information of user queries and chat records into high-dimensional semantic vectors; an image encoding submodule, using the CLIP visual encoder to extract features from visual content and generate semantic vectors that are aligned with the text; a speech encoding submodule, used to generate semantic vectors from speech information through an ASR conversion path or an audio feature extraction path; and an LLM submodule, used to convert unstructured multimodal information into structured semantics containing core elements.

[0053] Example 3: This invention provides a server, including at least one processor and a memory communicatively connected to the processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the processor to cause the at least one processor to perform the chat history multimodal search method based on CLIP and LLM provided in Embodiment 1 of this invention.

[0054] It should be understood that although the steps in the flowcharts of the various embodiments of this invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this invention, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, each embodiment...At least some steps in the embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be executed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0055] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program may be stored in a non-volatile computer-readable storage medium. When the program is executed, it may include the processes of the embodiments of the above methods. In the embodiments provided by the present invention, any reference to memory, storage, database or other media may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0056] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention. Specification page 9 / 9, CN 121434418 A, Figure 1, Figure 2; Specification drawing page 1 / 2, CN 121434418 A, Figure 3, Figure 4; Specification drawing page 2 / 2, CN 121434418 A. This invention belongs to the field of information retrieval technology, and provides a method and system.for multimodal search of chat history based on CLIP and LLM. It utilizes LLM to perform intent parsing and semantic enhancement on information in user queries and chat history. CLIP multimodal encoding unifies the text, image, and voice information in user queries and chat history into the same semantic space. Multimodal feature vectors are stored in a vector database, and dynamic updates and efficient retrieval are achieved based on a hierarchical index structure. The retrieval results are dynamically weighted and sorted according to semantic relevance, time decay weight, and user preference. Multimodal search results containing text, images, mixed text and images, or voice summaries are displayed to the user. This invention, through CLIP cross-modal semantic mapping and LLM intent completion, can capture deep semantic relationships, optimize semantic understanding, support natural language query parsing, improve the efficiency of multimodal information acquisition, and optimize theuser interaction experience. Abstract

Claims

1. A CLIP and LLM-based chat record multi-modal search method, characterized in that, The method comprises: Using LLM to analyze the intent and enhance the semantics of information in user queries and chat records, and then using CLIP multi-modal encoding to uniformly encode text, image, and voice information in user queries and chat records into the same semantic space; Storing multi-modal feature vectors into a vector database and realizing dynamic updating and efficient retrieval based on a hierarchical index structure; Based on a hybrid ranking strategy, dynamically weighting and ranking the retrieval results according to semantic relevance, time decay weight, and user preference; Showing multi-modal search results including text, image, mixed text and image, or voice summary to the user.

2. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, In the process of using LLM to analyze the intent and enhance the semantics of information in user queries and chat records, the input information is analyzed by LLM to complete the intent, and an enhanced vector containing time, entity, emotion, and topic is generated.

3. The CLIP and LLM based chat record multimodal search method of claim 1, wherein, The specific steps of the CLIP multi-modal encoding include: Using the Transformer architecture of CLIP to encode text information into semantic vectors; Using the CLIP visual encoder to extract visual feature vectors from image information; Generating semantic vectors from voice information through ASR conversion or audio feature extraction path.

4. The CLIP and LLM based chat record multimodal search method of claim 1, wherein, The dynamic updating and efficient retrieval based on the hierarchical index structure include: Building a hierarchical index structure containing time index and user / group index; Using incremental learning mechanism to encode new messages in real time and update the index; Using cache strategy for indexed data, and optimizing the cache strategy based on access frequency and LRU algorithm.

5. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, The scoring formula of the hybrid ranking strategy is: wherein, is a CLIP semantic relevance score, is a temporal decay weight coefficient, is a user preference coefficient, , , is a normalization weight coefficient.

6. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, The time decay weight coefficient is calculated using an exponential decay function, which is expressed as: wherein, is a message timestamp, is a current time, is a number of weights.

7. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, The user preference coefficient is generated by: Building short-term interest vectors and long-term preference vectors based on user historical behavior data; Adjusting the weight based on explicit feedback signals, which is expressed as: wherein, is a short-term interest vector, is a long-term preference vector, F is a set of explicit feedbacks, is a feedback weight coefficient.

8. A CLIP and LLM based chat record multimodal search system, characterized in that, The system comprises: LLM processing module, for analyzing the intent and enhancing the semantics of information in user queries and chat records, and converting unstructured multi-modal information into structured semantics containing core elements; CLIP multi-modal encoding module, for uniformly encoding text, image, and voice information in user queries and chat records into the same semantic space, and performing intent analysis and semantic enhancement based on large model LLM; Storage module, containing vector database and hierarchical index structure, for storing multi-modal feature vectors and supporting dynamic updating and efficient retrieval; Hybrid ranking strategy module, for dynamically weighting and fusing to generate comprehensive ranking scores based on semantic relevance, time decay weight, and user preference; User interaction module, for natural language query analysis and structured information extraction of user input, and for displaying multi-modal search results and voice summary information.

9. The CLIP and LLM based chat log multimodal search system of claim 8, wherein, The CLIP multi-modal encoding module includes: Text encoding submodule, based on the Transformer architecture of CLIP, for mapping text information in user queries and chat records into high-dimensional semantic vectors; Image encoding submodule, using CLIP visual encoder to extract features from visual content and generate semantic vectors aligned with text; The voice coding submodule is configured to generate a semantic vector from voice information through an ASR conversion path or an audio feature extraction path; The LLM submodule is configured to convert unstructured multi-modal information into structured semantics containing core elements.

10. A server, characterized by The device comprises at least one processor and a memory connected to the processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to perform the method for multi-modal search of chat records based on CLIP and LLM according to any one of claims 1 to 6.

Images