Chat record multi-modal search method and system based on CLIP and LLM

Abstract

The invention belongs to the technical field of information retrieval, and provides a chat record multi-modal search method and system based on CLIP and LLM, and the LLM is used for carrying out intention analysis and semantic enhancement on information in user query and chat records. Uniformly encoding texts, images and voice information in user query and chat records into the same semantic space through CLIP multi-mode encoding; the multi-modal feature vectors are stored in a vector database, and dynamic updating and efficient retrieval are achieved based on a hierarchical index structure; dynamic weighted sorting of semantic correlation, time decay weight and user preference is carried out on retrieval results; and displaying the multi-modal search result containing the text, the image, the image-text mixture or the voice abstract to the user. Through CLIP cross-modal semantic mapping and LLM intention completion, deep semantic association can be captured, semantic understanding can be optimized, natural language query analysis can be supported, multi-modal information acquisition efficiency can be improved, and user interaction experience can be optimized.

Description

Technical Field

[0001] This invention belongs to the field of information retrieval technology, specifically relating to a multimodal search method and system for chat history based on CLIP and LLM, applicable to information retrieval scenarios in various instant messaging software, online collaboration platforms, and enterprise-level communication tools. Background Technology

[0002] With the widespread use of instant messaging (IM) software, users generate a large amount of information during chats, including text messages, images, emoticons, voice messages, and files. This rich information constitutes complex multimodal data, posing challenges to users' information retrieval. In existing instant messaging systems, chat history search functions mainly rely on keyword matching technology. Users input keywords, and the system searches the text content of the chat history, returning records containing those keywords. However, this traditional search method has the following limitations: 1. Single-modal search limitation: Existing search functions mainly target text information and cannot effectively handle non-text content such as images, voice messages, and emoticons. 2. Insufficient semantic understanding: Keyword matching technology is based solely on direct character matching, lacking a deep understanding of semantics. 3. Timeliness issues: With chat history constantly being updated, traditional search systems typically need to periodically build indexes, failing to reflect the latest chat content in real time. This can lead to delays when users search for the latest messages, failing to meet the high timeliness requirements of instant messaging. 4. Insufficient personalization: Existing search functions typically employ a uniform search strategy and offer only a single search method, supporting only text search and failing to optimize based on users' personal preferences and usage habits.

[0003] Therefore, this invention proposes a multimodal search method and system for instant messaging chat records based on Contrastive Language-Image Pretraining (CLIP) and Large Language Model (LLM). By enhancing information through LLM, the CLIP model maps information from multiple modalities such as text and images to the same semantic space, thereby achieving cross-modal semantic matching. Summary of the Invention

[0004] The purpose of this invention is to overcome the existing defects and provide a multimodal search method and system for chat history based on CLIP and LLM, which can effectively improve the search efficiency and accuracy of chat history.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: The first objective of this invention is to provide a multimodal search method for chat history based on CLIP and LLM, comprising: LLM is used to perform intent parsing and semantic enhancement on information in user queries and chat history, and CLIP multimodal coding is used to encode the text, image and voice information in user queries and chat history into the same semantic space. Multimodal feature vectors are stored in a vector database, and dynamic updates and efficient retrieval are achieved based on a hierarchical index structure; The search results are dynamically weighted based on semantic relevance, time decay weight, and user preference using a hybrid ranking strategy. Display multimodal search results to users, including text, images, mixed text and images, or audio summaries.

[0006] Furthermore, in the process of using LLM to perform intent parsing and semantic enhancement on information in user queries and chat history, LLM is used to complete the intent of the input information and generate enhanced vectors containing time, entities, sentiment, and topics.

[0007] Furthermore, the specific steps of CLIP multimodal coding include: Text information is encoded into semantic vectors using CLIP's Transformer architecture; The CLIP visual encoder is used to extract visual feature vectors from the image information; Semantic vectors are generated from speech information through ASR conversion or audio feature extraction paths.

[0008] Furthermore, the implementation of dynamic updates and efficient retrieval based on a hierarchical index structure includes: Construct a hierarchical index structure that includes time indexes and user / group indexes; An incremental learning mechanism is used to encode new messages and update the index in real time; A caching strategy is adopted for indexed data, and the caching strategy is optimized based on access frequency and LRU algorithm.

[0009] Furthermore, the scoring formula for the hybrid sorting strategy is as follows: in, Score the semantic relevance of CLIP. This is the time decay weighting coefficient. User preference coefficient , , These are the normalized weighting coefficients.

[0010] Furthermore, the time decay weighting coefficient is calculated using an exponential decay function, expressed as: in, For message timestamps, This is the current time.

[0011] Furthermore, the user preference coefficient is generated in the following way: Construct short-term interest vectors and long-term preference vectors based on users' historical behavior data; Adjusting the weights using explicit feedback signals is expressed as follows: in, For short-term interest vectors, Let F be the long-term preference vector, and F be the set of explicit feedback. This refers to the feedback weighting coefficient.

[0012] Another objective of this invention is to provide a multimodal chat history search system based on CLIP and LLM, comprising: The LLM processing module is used to perform intent parsing and semantic enhancement on information in user queries and chat history, transforming unstructured multimodal information into structured semantics containing core elements. The CLIP multimodal coding module is used to encode text, image, and voice information from user queries and chat history into the same semantic space, and perform intent parsing and semantic enhancement based on the large model LLM. The storage module includes a vector database and a hierarchical index structure, used to store multimodal feature vectors and support dynamic updates and efficient retrieval; The hybrid ranking strategy module generates a comprehensive ranking score by dynamically weighting and fusing semantic relevance, time decay weight, and user preferences. The user interaction module is used to perform natural language query parsing and structured information extraction on user input, and to display multimodal search results and speech summary information.

[0013] Furthermore, the CLIP multimodal coding module includes: The text encoding submodule, based on CLIP's Transformer architecture, is used to map text information from user queries and chat logs into high-dimensional semantic vectors. The image encoding submodule uses the CLIP visual encoder to extract features from the visual content and generate semantic vectors aligned with the text. The speech coding submodule is used to generate semantic vectors from speech information through ASR conversion path or audio feature extraction path; Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the multimodal search method for chat history based on CLIP and LLM provided by the first object of the present invention.

[0014] Another object of the present invention is to provide a server comprising at least one processor and a memory communicatively connected to the processor, the memory storing instructions executable by the at least one processor, the instructions being executed by the processor to cause the at least one processor to perform the CLIP and LLM-based multimodal chat history search method provided in the first object of the present invention.

[0015] In combination with the above technical solutions, the beneficial effects of the present invention compared with the prior art are as follows: This invention replaces keyword character matching with CLIP cross-modal semantic mapping and LLM intent completion to capture deep semantic relationships and improve the problem of insufficient semantic understanding. Through the CLIP multimodal coding module, this invention maps text, images, and speech (encoded via ASR or direct encoding) to the same semantic space. Combined with LLM intent completion, it achieves deep semantic alignment, overcoming the pain points of "single-modal limitations" and "insufficient semantic understanding." It also solves the problem that existing technologies often use separate encoding for text, images, and speech (e.g., BERT for text and Wav2Vec for speech), resulting in weak cross-modal semantic relationships and only surface-level matching.

[0016] This invention uses a vector database to uniformly store multimodal vectors, designs a hierarchical index of "time + user / group", and combines incremental learning to update the index in real time with intelligent caching. This breaks down storage barriers and achieves millisecond-level response and real-time retrieval. It solves the problems of existing technologies that use heterogeneous storage of multimodal data (text stored in databases, images stored in file systems), require cross-system retrieval and rely on periodic indexing, resulting in low efficiency and poor timeliness.

[0017] This invention dynamically weights and fuses CLIP semantic similarity, time decay weights (exponential function), and user preferences (short-term LSTM + long-term aggregation + explicit feedback), with weights learned and adjusted online. It balances semantic accuracy, timeliness, and personalization, outperforming traditional fixed strategies. It also solves the problem that traditional technologies often use fixed weight ranking (such as only by time or keyword matching degree) and lack semantic and personalized adaptation.

[0018] This invention supports natural language query parsing (extracting structured information such as time and people), provides mixed text and image sorting, image search, and voice summarization (highlighting key segments), improving the efficiency of multimodal information acquisition and optimizing the user interaction experience. It overcomes the limitations of existing technologies' single-modal search, supporting multimodal content retrieval including text, images, voice, and emojis, breaking through the limitation of only processing text, and addressing the problems of existing technologies' single interaction methods (such as text-only search) and fragmented multimodal display. Attached Figure Description

[0019] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is the flowchart of the multimodal chat history search method based on CLIP and LLM provided in the embodiments of the present invention; Figure 2 This is a schematic diagram of the method provided in the embodiment of the present invention; Figure 3 This is a flowchart of user interaction provided in an embodiment of the present invention; Figure 4 This is a structural block diagram of the chat history multimodal search system based on CLIP and LLM provided in an embodiment of the present invention. Detailed Implementation

[0020] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0021] Example 1: like Figure 1 The image shows an embodiment of the multimodal chat history search method based on CLIP and LLM provided by the present invention, which specifically includes the following steps: S1: Use LLM to perform intent parsing and semantic enhancement on information in user queries and chat history, and then use CLIP multimodal coding to encode the text, image and voice information in user queries and chat history into the same semantic space. S2: Store multimodal feature vectors in a vector database and achieve dynamic updates and efficient retrieval based on a hierarchical index structure; S3: Based on a hybrid ranking strategy, the search results are dynamically weighted according to semantic relevance, time decay weight, and user preference. S4: Display multimodal search results to users, including text, images, mixed text and images, or audio summaries.

[0022] Specifically, such as Figure 2As shown, this embodiment of the invention utilizes the CLIP multimodal encoding step, using CLIP's text encoder (such as Transformer) to map text information from user queries and chat logs into semantic vectors; and uses CLIP's visual encoder to extract features from images, emoticons, and video keyframes to generate visual semantic vectors. For speech encoding, it first converts to text using ASR (such as Whisper) and then feeds it into the CLIP text encoder, or combines audio features (such as VGGish) for cross-modal alignment. Based on CLIP, LLM (such as ChatGLM) is introduced to perform intent parsing on the input, generating enhanced vectors containing information such as time, entities, sentiment, and topics. A vector database stores the multimodal feature vectors of all chat logs, supporting efficient similarity retrieval. Multiple strategies are combined for sorting, and the search result with the highest score is returned. LLM is used to perform multi-round semantic parsing of queries, supporting image, natural language, and speech queries, and parsing structured information such as time and people. Multimodal search results are displayed, such as: mixed image and text sorting (related images + contextual text), speech summaries, etc.

[0023] The various steps of the present invention will be described in detail below.

[0024] I. CLIP Multimodal Coding Steps: This step mainly consists of text encoding, speech encoding, image encoding, and LLM processing, among which image encoding can encode video. Existing technologies rely on keyword character matching and lack semantic understanding capabilities (e.g., they cannot recognize the semantic association between "meeting" and "conference"). This invention, through CLIP's cross-modal semantic mapping and LLM's intent completion, can not only understand the deep semantics of text but also transform the content of images and speech into associative semantic vectors, significantly improving the semantic accuracy of cross-modal matching.

[0025] 1. LLM processing: Simultaneously processing content with LLM for intent completion transforms unstructured multimodal information (speech, text, images) into structured semantics containing core elements such as time, people, and events. This enriches the semantic dimension of the original information and provides a unified input format for the CLIP model's cross-modal encoding. This approach allows information from different modalities to directly calculate their relevance in the same semantic space using cosine similarity. For example, when a user inputs the text query "team activities next week," the system can simultaneously match related audio recordings (e.g., "meeting next Monday"), text messages (e.g., "discussing projects on Wednesday"), and image posters (e.g., "dinner on Friday"), significantly improving the accuracy and semantic understanding depth of cross-modal search.

[0026] 2. Text encoding: The text encoding process of this invention is based on the CLIP Transformer architecture and is used to map text information such as user queries and chat logs into high-dimensional semantic vectors. The specific implementation includes the following steps: Input preprocessing involves standardizing the original text (e.g., Unicode normalization, special character filtering) and converting the text into a token sequence using sub-word segmentation (BPE algorithm). Embedding representation maps tokens to vectors using a learnable word embedding matrix (dimension d×|V|, where d is the hidden layer dimension and |V| is the vocabulary size), and overlays positional encodings (sine / learned) to preserve sequence information; Feature extraction employs a multi-layer Transformer encoder (e.g., 12 layers) for context modeling, with each layer containing a multi-head self-attention (MHA) and a feedforward network (FFN). Semantic aggregation generates fixed-dimensional text semantic vectors (e.g., 512-dimensional) through [CLS] tagging or mean pooling.

[0027] 3. Image encoding: The image encoding process of this invention employs a CLIP visual encoder (ViT or ResNet variant), supporting feature extraction from visual content such as images, emojis, and video keyframes to generate semantic vectors aligned with text. The specific process includes: Image preprocessing: The input image is normalized to a fixed resolution (e.g., 224×224) and the channels are standardized (mean / variance adjustment); Feature extraction: Taking the ViT architecture as an example: the image is segmented into a 16×16 Patch sequence, linearly projected and then superimposed with position encoding, and global features are extracted by the Transformer encoder; Output mapping: Visual features are aligned to the text semantic space through a projection layer. The module supports a dynamic attention mechanism to enhance the feature weights of key regions.

[0028] 4. Voice encoding: The speech encoding process of this invention provides a dual-path scheme to achieve efficient mapping from speech to semantic vectors: (1) ASR text conversion path: The Whisper model is used for speech recognition (ASR), which converts the audio signal SS into text TT, and then processes it through the CLIP text encoder to support multilingual transcription and text post-processing (such as punctuation restoration and spoken language standardization).

[0029] (2) Direct encoding path of audio features: Extract Log-Mel spectrogram, extract high-level acoustic features through VGGish network, and then align to CLIP space through cross-modal projection layer.

[0030] II. Vector Storage: This process comprises a hierarchical vector database storage, a hierarchical index structure, a dynamic index update module, and an intelligent caching strategy. Its core lies in achieving highly timely semantic retrieval through a multi-level index structure and incremental learning mechanism, while simultaneously utilizing session isolation strategies and caching optimization algorithms to reduce cross-contextual interference and improve response speed. Unlike existing technologies, it employs a vector database to uniformly store multimodal feature vectors, breaking down the barriers of heterogeneous multimodal data storage and achieving unified and efficient cross-modal retrieval; it innovatively designs a hierarchical structure of "time index + user / group index," balancing time-range retrieval with session independence to improve retrieval accuracy; it processes new messages and updates the index in real time through an incremental learning mechanism, solving the time lag problem of traditional batch updates; and it uses an intelligent caching strategy based on access frequency and the LRU algorithm to specifically optimize response speed for multimodal retrieval scenarios. These designs collectively constitute a unique innovation in storage and retrieval mechanisms.

[0031] 1. Vector database storage: A vector database is used to store the multimodal feature vectors of all chat records. This vector database efficiently processes high-dimensional vector data, supports near-nearest neighbor retrieval, and achieves millisecond-level response times. A deep learning model is used to convert multimodal data such as text, images, and audio from chat records into high-dimensional vectors, which are then uniformly stored in the vector database.

[0032] 2. Hierarchical index structure: Time Index: Messages are stored in layers based on timestamps, supporting time range retrieval. Chat history is categorized by timestamps, with each layer corresponding to a time interval, facilitating quick location of chat records within a specific time range.

[0033] User / Group Index: Categorized by session context to avoid cross-session interference. Chat logs are stored by user or group to ensure data independence between different sessions and improve retrieval accuracy.

[0034] 3. Dynamic Index Update Module: An incremental learning mechanism is adopted to encode and update the index in real time when a new message is generated, ensuring the timeliness of search results. The specific process is as follows: (1) When a new message arrives, its multimodal feature vector is extracted through a pre-trained model. (2) The new vector is inserted into the vector database in real time, and the index structure is updated. (3) Through a dynamic index update mechanism, the index in the database is always up-to-date, thereby providing the latest search results.

[0035] 4. Caching strategy: A caching strategy is adopted for indexed data, with high-frequency access content loaded first to improve response speed. The specific implementation is as follows: (1) The data is sorted by access frequency, and the chat records and their indexes that are accessed frequently are loaded into the cache; (2) The cache adopts the LRU (Least Recently Used) algorithm to periodically evict low-frequency access data to ensure effective utilization of cache space. III. Hybrid sorting strategy: This strategy generates a comprehensive ranking score by dynamically weighting and fusing semantic relevance scores, time decay weights, and user preference coefficients. Its core formula is defined as follows: in, The normalized weight coefficients are dynamically adjusted through online learning. Through CLIP-driven cross-modal semantic alignment, dynamic weight adjustment through online learning, refined time decay design, dynamically adapted user preference modeling, and a closed-loop feedback mechanism, it significantly differs from traditional solutions such as discrete modality encoding, fixed weight strategies, and complex interest genome modeling, achieving innovation and improvement in semantic matching accuracy, scene adaptability, and personalized response speed.

[0036] 1. Semantic relevance score: Semantic relevance scoring is a core component of the hybrid ranking strategy. This scoring mechanism is based on the CLIP model, and... This method measures content relevance by calculating the similarity between query vectors and chat history vectors. It delves into the semantic relationship between queries and content, ensuring search results are highly semantically aligned with user needs. This allows the system to prioritize content semantically closest to the user's query, improving the accuracy and relevance of search results. Compared to traditional methods that use discrete encodings like BERT, Wav2Vec, and visual models, mapped to a shared space via cross-modal adversarial networks, this approach more directly captures the deep semantic connections between cross-modal content, reducing information loss during modality transformation.

[0037] 2. Time weighting: Time weighting is another important scoring dimension in hybrid ranking strategies. This mechanism weights content based on the message's write time, ensuring that more recent content has a higher weight. This design effectively improves the timeliness of search results, allowing users to quickly access the latest information. In scenarios where information updates rapidly, such as news and social media, the introduction of time weighting ensures that users don't miss important updates, thereby enhancing the usability and attractiveness of the search system.

[0038] Based on message timestamp With current time Construct an exponential decay function based on the interval: 3. User preferences: User preferences are a key aspect of personalization in hybrid ranking strategies. This mechanism learns personalized weights for each user based on their historical click behavior and interaction data through machine learning algorithms. The system can dynamically adjust the ranking of search results according to user interests and preferences, thus providing a tailored search experience for each user. This personalization strategy offers greater dynamism, flexibility, and optimization capabilities, enabling it to more accurately capture changes in user interests and improve search accuracy.

[0039] The model distinguishes between short-term interests and long-term preferences by analyzing users' historical behavioral data. (1) Short-term interests: Based on the user's most recent k interactions, capture the user's current immediate interests. Let... Given the set of content from the user's k most recent interactions, the model encodes these interactions using an LSTM network to obtain short-term interest vectors. (2) Long-term preferences: Based on users' historical behavioral data over the past T days, we can mine stable preference patterns. Let... Given the set of content the user has interacted with in the last T days, the model aggregates this content to obtain a long-term preference vector. .

[0040] The model also incorporates explicit user feedback (such as "irrelevant" labels) for real-time adjustments. Let F be the set of explicit user feedback; the model adjusts its weights based on these feedback signals. ; in, These are the weighting coefficients of the feedback signal, used to balance the impact of explicit feedback on the user preference vector. For example, when a user marks recommended content as "irrelevant," the model immediately adjusts the weights of related topics, reducing their priority in subsequent recommendations. This dynamic adjustment mechanism allows the model to more flexibly capture dynamic changes in user interests, offering greater adaptability and flexibility compared to constructing user interest profiles through interest genomes and models.

[0041] IV. User Interaction: The user interaction process, by introducing LLM and multimodal search technologies, achieves functions such as natural language query parsing, mixed text and image sorting, image search, and voice summarization. These functions not only enhance the user experience but also make the search process more efficient and convenient, meeting the diverse needs of users in different scenarios. Compared to traditional solutions, introducing LLM technology into the search process enhances the system's understanding of the retrieved information, while the model uses CLIP encoding to achieve image search functionality. The user interaction module provides rich multimodal search result display methods to meet the needs of different users. The specific steps are as follows: 1. User Submission of Query: Users can input their query content via natural language text, images, or voice. The system supports multimodal input, including text descriptions, uploaded images, or voice commands. For voice queries, the system will first use speech recognition technology to convert it into text (voice summarization function) for subsequent processing.

[0042] 2. Query Parsing and Time Detection: The system parses user queries and checks if they contain time-related information (such as date or time range). If the query contains time, the time range is extracted; otherwise, keyword processing begins. Simultaneously, a global time-domain search is performed to ensure that time-related queries cover all possible time dimensions.

[0043] 3. Multimodal Coding and LLM Enhanced Understanding: For text queries, LLM is used for deep semantic analysis to enhance the understanding of query intent (e.g., natural language query parsing). The CLIP text encoder then converts the query text into a vector representation to further enhance the understanding of query intent (e.g., natural language query parsing). For image queries (image search), LLM is used for contextual understanding to achieve intelligent interpretation of image content. The CLIP visual encoder then converts the uploaded image into a vector representation. This step ensures the system can handle mixed text and image queries and provides a unified vector representation for subsequent retrieval.

[0044] 4. Vector Database Retrieval: The encoded query vector (text or image vector) is input into a vector database for similarity retrieval. The database stores pre-encoded vectors of documents, images, or other multimedia content. The retrieval process returns a preliminary set of candidate matching results.

[0045] 5. Applying Time Decay Weights: If the query includes a time range, the system applies time decay weights to the search results. This means that newer content will receive higher weights, while older content will receive lower weights, to ensure that the search results match the time relevance (a refinement of global time-domain search).

[0046] 6. Calculate the similarity score: Based on the query vector and candidate result vector, calculate the similarity score for each result. The score reflects the degree of semantic matching between the content and the query.

[0047] 7. Load Personalized Preferences: The system loads the user's personalized preferences (such as historical search behavior, interest tags, etc.) and adjusts the similarity score. This step makes the search results more in line with the user's individual needs and improves the personalized experience.

[0048] 8. Generate the final ranking: Combine similarity scores, time decay weights, and personalized preferences to comprehensively rank all candidate results. The ranking algorithm ensures that mixed text and image results (such as mixed text and image ranking) are optimized based on relevance, timeliness, and user preferences.

[0049] 9. Return topk results: The system returns the sorted topk results to the user. The results may be displayed in various formats, such as lists, grids, or summaries, to meet the needs of different users (such as multimodal search results display).

[0050] This user interaction module, by incorporating LLM and multimodal technologies, constructs an intelligent and efficient search system. Its core lies in leveraging the powerful natural language understanding capabilities of LLM to accurately parse user queries (such as "find the product introduction Zhang San mentioned last week") and automatically extract key structured information such as time and people, thereby deeply understanding the user's intent. The system supports multimodal search, not only sorting and displaying results in a mixed text and image format but also enabling "image search" through CLIP encoding. Furthermore, for audio content, the module provides a voice summarization function, using LLM analysis to convert the audio into text and highlight relevant segments, allowing users to quickly locate key information. Through a series of functions including natural language parsing, mixed text and image sorting, image search, and voice summarization, this system significantly improves the accuracy, richness, and efficiency of searches, providing users with a more intuitive and convenient cross-modal information retrieval experience.

[0051] Example 2: like Figure 4 As shown, this embodiment of the invention provides a multimodal chat history search system based on CLIP and LLM, comprising: The LLM processing module is used to perform intent parsing and semantic enhancement on information in user queries and chat history, transforming unstructured multimodal information into structured semantics containing core elements. The CLIP multimodal coding module is used to encode text, image, and voice information from user queries and chat history into the same semantic space, and perform intent parsing and semantic enhancement based on the large model LLM. The storage module includes a vector database and a hierarchical index structure, used to store multimodal feature vectors and support dynamic updates and efficient retrieval; The hybrid ranking strategy module generates a comprehensive ranking score by dynamically weighting and fusing semantic relevance, time decay weight, and user preferences. The user interaction module is used to perform natural language query parsing and structured information extraction on user input, and to display multimodal search results and speech summary information.

[0052] Preferably, the CLIP multimodal coding module in this embodiment of the invention includes: The text encoding submodule, based on CLIP's Transformer architecture, is used to map text information from user queries and chat logs into high-dimensional semantic vectors. The image encoding submodule uses the CLIP visual encoder to extract features from the visual content and generate semantic vectors aligned with the text. The speech coding submodule is used to generate semantic vectors from speech information through ASR conversion path or audio feature extraction path; The LLM submodule is used to transform unstructured multimodal information into structured semantics that includes core elements.

[0053] Example 3: This invention provides a server, including at least one processor and a memory communicatively connected to the processor. The memory stores instructions executable by the at least one processor. The instructions are executed by the processor to cause the at least one processor to perform the chat history multimodal search method based on CLIP and LLM provided in Embodiment 1 of this invention.

[0054] It should be understood that although the steps in the flowcharts of the various embodiments of the present invention are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in the present invention, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least some steps in the various embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0055] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0056] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A CLIP and LLM-based chat record multi-modal search method, characterized in that, The method comprises: Using LLM to analyze the intent and enhance the semantics of information in user queries and chat records, and then using CLIP multi-modal encoding to uniformly encode text, image, and voice information in user queries and chat records into the same semantic space; Storing multi-modal feature vectors into a vector database and realizing dynamic updating and efficient retrieval based on a hierarchical index structure; Based on a hybrid ranking strategy, dynamically weighting and ranking the retrieval results according to semantic relevance, time decay weight, and user preference; Showing multi-modal search results including text, image, mixed text and image, or voice summary to the user.

2. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, In the process of using LLM to analyze the intent and enhance the semantics of information in user queries and chat records, the input information is analyzed by LLM to complete the intent, and an enhanced vector containing time, entity, emotion, and topic is generated.

3. The CLIP and LLM based chat record multimodal search method of claim 1, wherein, The specific steps of the CLIP multi-modal encoding include: Using the Transformer architecture of CLIP to encode text information into semantic vectors; Using the CLIP visual encoder to extract visual feature vectors from image information; Generating semantic vectors from voice information through ASR conversion or audio feature extraction path.

4. The CLIP and LLM based chat record multimodal search method of claim 1, wherein, The dynamic updating and efficient retrieval based on the hierarchical index structure include: Building a hierarchical index structure containing time index and user / group index; Using incremental learning mechanism to encode new messages in real time and update the index; Using cache strategy for indexed data, and optimizing the cache strategy based on access frequency and LRU algorithm.

5. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, The scoring formula of the hybrid ranking strategy is: wherein, is a CLIP semantic relevance score, is a temporal decay weight coefficient, is a user preference coefficient, , , is a normalization weight coefficient.

6. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, The time decay weight coefficient is calculated using an exponential decay function, which is expressed as: wherein, is a message timestamp, is a current time, is a number of weights.

7. The CLIP and LLM based chat record multimodal search method according to claim 1, characterized in that, The user preference coefficient is generated by: Building short-term interest vectors and long-term preference vectors based on user historical behavior data; Adjusting the weight based on explicit feedback signals, which is expressed as: wherein, is a short-term interest vector, is a long-term preference vector, F is a set of explicit feedbacks, is a feedback weight coefficient.

8. A CLIP and LLM based chat record multimodal search system, characterized in that, The system comprises: LLM processing module, for analyzing the intent and enhancing the semantics of information in user queries and chat records, and converting unstructured multi-modal information into structured semantics containing core elements; CLIP multi-modal encoding module, for uniformly encoding text, image, and voice information in user queries and chat records into the same semantic space, and performing intent analysis and semantic enhancement based on large model LLM; Storage module, containing vector database and hierarchical index structure, for storing multi-modal feature vectors and supporting dynamic updating and efficient retrieval; Hybrid ranking strategy module, for dynamically weighting and fusing to generate comprehensive ranking scores based on semantic relevance, time decay weight, and user preference; User interaction module, for natural language query analysis and structured information extraction of user input, and for displaying multi-modal search results and voice summary information.

9. The CLIP and LLM based chat log multimodal search system of claim 8, wherein, The CLIP multi-modal encoding module includes: Text encoding submodule, based on the Transformer architecture of CLIP, for mapping text information in user queries and chat records into high-dimensional semantic vectors; Image encoding submodule, using CLIP visual encoder to extract features from visual content and generate semantic vectors aligned with text; The voice coding submodule is configured to generate a semantic vector from voice information through an ASR conversion path or an audio feature extraction path; The LLM submodule is configured to convert unstructured multi-modal information into structured semantics containing core elements.

10. A server, characterized by The device comprises at least one processor and a memory connected to the processor in communication, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the processor to enable the at least one processor to perform the method for multi-modal search of chat records based on CLIP and LLM according to any one of claims 1 to 6.

Images