Self-adaptive multi-modal retrieval system based on public opinion knowledge base updating
By constructing the RAG-Elasticsearch adaptive multimodal retrieval system, the problems of real-time updating and unified feature representation of public opinion data were solved, realizing efficient and accurate multimodal retrieval in government public opinion monitoring, and improving the timeliness and adaptability of the retrieval system.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NANJING LES CYBERSECURITY & INFORMATION TECH RES INST CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-05-01
AI Technical Summary
Existing multimodal retrieval systems suffer from problems such as outdated knowledge base, poor modality adaptability, and an imbalance between retrieval accuracy and efficiency in public opinion scenarios. They cannot achieve real-time dynamic updates of public opinion data and unified structured feature representation, resulting in delayed retrieval results and an inability to balance accuracy and efficiency.
An adaptive multimodal retrieval system based on RAG-Elasticsearch is constructed. The system performs unified format conversion and feature extraction through a multimodal data access module, achieves real-time updates by combining a dynamic update module for the public opinion knowledge base, uses an Elasticsearch enhanced retrieval module for collaborative retrieval, and generates the final answer through a RAG retrieval enhancement generation module.
It enables real-time dynamic updates of the public opinion knowledge base, improves the accuracy and efficiency of multimodal data retrieval, ensures the timeliness and accuracy of retrieval results, and meets the multimodal data needs of government public opinion monitoring.
Smart Images
Figure CN121960744A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of information retrieval and public opinion analysis technology, and in particular to an adaptive multimodal retrieval system based on updates to a public opinion knowledge base. Background Technology
[0002] With the explosive growth of internet public opinion data, this data exhibits a multimodal mix of text, images, and videos. Existing multimodal retrieval systems suffer from the following core deficiencies in public opinion scenarios: 1) Lagging knowledge base. Existing systems mostly use static knowledge bases, lacking a dynamic update mechanism for the "time-sensitive" nature of public opinion data. This makes it impossible to promptly remove outdated public opinion information or supplement real-time public opinion content, resulting in delayed public opinion analysis results; 2) Poor modality adaptability. The feature extraction logic for various types of public opinion data, such as text, images, and videos, is fragmented, failing to achieve a unified structured feature representation and thus failing to efficiently adapt to multimodal retrieval needs; 3) Imbalance between retrieval accuracy and efficiency. Traditional CPU-based Elasticsearch (ES) retrieval solutions have low performance, and the synergy between Retrieval-Augmented Generation (RAG) technology and ES retrieval is insufficient—retrieval results are not deeply integrated with real-time information in the knowledge base, making it difficult to balance the accuracy of generated answers with retrieval efficiency. Summary of the Invention
[0003] Purpose of the invention: To address the shortcomings of existing technologies, this invention solves the following problems: 1) Realizes real-time dynamic updates of the public opinion knowledge base, solving the problem of knowledge base lag; 2) Constructs a unified structured feature representation of multimodal public opinion data to improve modality adaptability; 3) Achieves deep collaboration between RAG and ES to balance the retrieval accuracy and efficiency of multimodal public opinion data.
[0004] This invention specifically provides a RAG-Elasticsearch (Elasticsearch Retrieval-Augmented Generation) adaptive multimodal retrieval system based on an updated public opinion knowledge base. This method specifically relates to the technical fields of multimodal data processing, Retrieval-Augmented Generation (RAG), and intelligent retrieval systems that collaborate with Elasticsearch, and is particularly suitable for efficient retrieval and accurate response to multimodal public opinion data in government public opinion monitoring scenarios.
[0005] The system of this invention includes a multimodal data access module, a dynamic update module for the public opinion knowledge base, an enhanced Elasticsearch retrieval module, and a RAG retrieval enhancement generation module;
[0006] The multimodal data access module is used to perform unified format conversion and feature extraction on text, image, and video public opinion data to generate structured modal feature vectors.
[0007] The dynamic update module of the public opinion knowledge base is used to realize the real-time update and version management of the public opinion knowledge base through incremental crawling, semantic deduplication and timeliness filtering operations.
[0008] The Elasticsearch enhanced retrieval module is used to construct a multi-dimensional vector index and perform a collaborative retrieval of vector retrieval and text retrieval based on the structured modal feature vector to obtain keyword and semantic comprehensive retrieval results;
[0009] The RAG retrieval enhancement generation module is used to integrate the keyword and semantic comprehensive retrieval results with the public opinion knowledge base information to form a Prompt, and send the Prompt to the LLM large language model to generate the target answer.
[0010] The multimodal data access module processes public opinion data that includes two types: text, images, and videos.
[0011] The timeliness filtering operation is used to remove historical public opinion data from the public opinion knowledge base that exceeds the preset timeliness threshold.
[0012] The Elasticsearch enhanced retrieval module includes distributed data nodes and distributed machine learning embedding nodes; wherein, the distributed machine learning embedding nodes are used to generate retrieval vectors based on the structured modality feature vectors, and the distributed data nodes are used to perform collaborative retrieval operations of vector retrieval and text retrieval.
[0013] The distributed machine learning embedding node supports the uploading and deployment of business-tuned models or industry-customized models.
[0014] The RAG search enhancement generation module is used to associate and integrate the keyword and semantic comprehensive search results with the corresponding real-time updated public opinion information in the public opinion knowledge base.
[0015] The system is used for monitoring public opinion in government affairs.
[0016] The system performs the following steps:
[0017] Step 1: The multimodal data access module performs unified processing on text, image, and video public opinion data, including:
[0018] The Sentence-BERT pre-trained sentence encoding model is used to extract the semantic vector v of each sentence for text-based public opinion analysis. text :
[0019] V text =Sentence-BERT(si ),
[0020] Among them, s i Given the input text, Sentence-BERT is a pre-trained text sentence feature extraction network;
[0021] High-level semantic features V of images are extracted using the multimodal pre-trained model CLIP. img :
[0022] V img = CLIP(I),
[0023] Where I is the input image;
[0024] After extracting keyframes from the video, image feature extraction is performed on each frame, and then the average value of the frame feature sequence is taken to obtain the video feature vector V. video :
[0025]
[0026] Among them, I i Let k be the i-th keyframe in the video, and k be the number of keyframes.
[0027] Multimodal feature fusion generates a structured modal feature vector V. multi :
[0028] V multi =αV text +βV img +γV video
[0029] Where α, β, γ are modal weights, and α + β + γ = 1;
[0030] Step 2, the public opinion knowledge base dynamic update module executes the following process:
[0031] Step 2-1: Use web crawlers to capture text, image, and video data from publicly available public opinion platforms in real time, and access multimodal data;
[0032] Step 2-2: Calculate the feature vector similarity between the newly added data and the existing data in the knowledge base, using V... new and V old cosine similarity Sim(V) new V old As a feature vector similarity:
[0033]
[0034] Where Sim(V) new V old)∈[0,1], representing the similarity between the old and new knowledge bases. If the similarity is greater than the preset threshold τ, it is judged as duplicate data and removed; V new and V old Representing the vectors of the old and new knowledge bases respectively, ||V new ||and||V old || represent the L2 norms of the old and new knowledge base vectors, respectively.
[0035] Step 2-2: Mark each piece of public opinion data with a timestamp, periodically delete data whose timestamps exceed the preset time window, and complete the knowledge base version update;
[0036] Step 3, enhance Elasticsearch retrieval;
[0037] Step 4: RAG retrieval enhancement generation.
[0038] Step 3 includes: The Elasticsearch Enhanced Search module performs the following process based on the distributed ES cluster:
[0039] Step 3-1, Vector Index Construction: Construct the structured modal feature vector V multi Write to Elasticsearch distributed data nodes to build a multi-dimensional vector index;
[0040] Step 3-2, Collaborative Retrieval: After receiving the user's retrieval request, the query vector V is first generated through the Elasticsearch distributed machine learning embedding nodes. q Then, vector retrieval and text retrieval are performed simultaneously; the vector retrieval involves matching V in the vector index. q The results of the cosine similarity Top-N; the text retrieval is an ES Boolean query and OR non-full-text search performed on the user-requested keywords;
[0041] Step 3-3, Result Fusion: Combine the vector retrieval and text retrieval results according to their weights to obtain the comprehensive retrieval score result based on keywords and semantics.
[0042] Step 4 includes: The RAG retrieval enhancement generation module performs the following procedures:
[0043] Step 4-1, Information Fusion: Associate the comprehensive search results with the corresponding real-time public opinion information in the knowledge base to extract the core content;
[0044] Step 4-2, Prompt assembly: Assemble the Prompt according to the format of search results, knowledge base information, and task instructions;
[0045] Step 4-3: Send the Prompt to the LLM large language model to generate the final public opinion analysis answer.
[0046] This invention offers the following advantages: 1) Improved timeliness. Through dynamic updates to the public opinion knowledge base and a distributed architecture mechanism, it enables rapid access and expiration of real-time public opinion information; 2) Enhanced multimodal adaptability. It unifies the feature representation of various types of public opinion data, supporting collaborative retrieval of text, images, and videos; 3) Balance between accuracy and efficiency. Deep collaboration between RAG and ES ensures both the accuracy of search results and improves search efficiency through the distributed architecture of ES. Attached Figure Description
[0047] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.
[0048] Figure 1 This is the architecture diagram of the RAG-Elasticsearch multimodal retrieval system updated to the public opinion knowledge base.
[0049] Figure 2 This is the workflow of the RAG retrieval system based on LangChain.
[0050] Figure 3 This is a sample of RAG-Elasticsearch text retrieval based on a public opinion knowledge base.
[0051] Figure 4 This is a sample of RAG-Elasticsearch image / video retrieval based on a public opinion knowledge base. Detailed Implementation
[0052] The following is in conjunction with the appendix Figure 1 This section details the workflow of the RAG-Elasticsearch adaptive multimodal retrieval system, which is updated based on a public opinion knowledge base.
[0053] Step 1: Multimodal data access and feature fusion extraction;
[0054] The multimodal data access module performs unified processing of text, image, and video public opinion data.
[0055] (1) Text data processing
[0056] The semantic vector of each sentence in the text-based public opinion text is extracted using the Sentence-BERT pre-trained sentence encoding model:
[0057] V text =Sentence-BERT(s i )
[0058] Among them, s i The input text is Sentence-BERT, a pre-trained text sentence feature extraction network.
[0059] (2) Image data processing
[0060] High-level semantic features of images are extracted using the multimodal pre-trained model CLIP.
[0061] V img = CLIP(I)
[0062] Where I is the input image and CLIP is a pre-trained image feature extraction network.
[0063] (3) Video data processing
[0064] After extracting keyframes from the video, image feature extraction is performed on each frame, and then the average value of the frame feature sequence is taken to obtain the video feature vector:
[0065]
[0066] Among them, I i Let k be the i-th keyframe in the video, and k be the number of keyframes.
[0067] (4) Multimodal feature fusion
[0068] V multi =αV text +βV img +γV video
[0069] Where α, β, and γ are modal weights, and α + β + γ = 1. These can be adaptively adjusted according to the public opinion scenario.
[0070] Step 2: Dynamically update the public opinion knowledge base
[0071] The dynamic update module of the public opinion knowledge base executes the following process:
[0072] (1) Incremental crawling
[0073] (2) Semantic deduplication
[0074] Calculate the similarity between the feature vectors of newly added data and existing data in the knowledge base, using cosine similarity:
[0075]
[0076] If the similarity is greater than the preset threshold τ, it is determined to be duplicate data and removed.
[0077] (3) Timeliness screening
[0078] Each piece of public opinion data is timestamped, and data whose timestamps exceed the preset time window (such as days / weeks / months / years) is deleted periodically to complete the knowledge base version update.
[0079] Step 3: Enhance Elasticsearch retrieval
[0080] The Elasticsearch Enhanced Search module performs the following process based on a distributed cloud Elasticsearch cluster:
[0081] (1) Vector index construction
[0082] The multimodal structured feature vector V multi Write to Elasticsearch distributed data nodes to build a multi-dimensional vector index.
[0083] (2) Collaborative retrieval
[0084] After receiving a user's search request, the query vector V is first generated through Elasticsearch's distributed machine learning embedding nodes. q Then simultaneously perform vector retrieval (matching V in the vector index). q The results of cosine similarity Top-N are used for text retrieval (ES Boolean query and OR non-full-text search are performed on the user-requested keywords).
[0085] (3) Results fusion
[0086] The results of vector retrieval and text retrieval are combined according to weights to obtain the keyword / semantic comprehensive retrieval score.
[0087] Step 4: RAG Search Enhancement Generation
[0088] The RAG retrieval enhancement generation module performs the following process:
[0089] (1) Information fusion
[0090] The comprehensive search results are linked with the corresponding real-time public opinion information in the knowledge base to extract core content (such as the time, subject, and viewpoint of the public opinion event).
[0091] (2) Prompt splicing
[0092] Compile the Prompt in the format of "Search Results + Knowledge Base Information + Task Instructions", as shown in the example:
[0093] "Given public opinion information: {core content of comprehensive search results}, and real-time supplementation from the knowledge base: {corresponding real-time public opinion details}, please generate a summary analysis of this public opinion event."
[0094] (3) LLM generates answers
[0095] The prompt is sent to the LLM large language model to generate the final public opinion analysis answer.
[0096] Example: In the implementation phase of this invention, LangChain was selected as the development support tool for large language model applications. Relying on the rich components and practical tools integrated by LangChain, developers can easily achieve interconnection and integration between large language models and external data sources. This framework can adapt to various application scenarios such as document parsing, content extraction, intelligent chatbot construction, and customized retrieval, significantly simplifying the development process of large language model-related applications and effectively reducing the development threshold and implementation complexity.
[0097] Leveraging the various components and toolsets integrated within the LangChain framework, developers can efficiently and conveniently build RAG retrieval enhancement systems. The basic workflow of this system is as follows: Figure 2 As shown.
[0098] 1. Knowledge base question-answering pair construction
[0099] In the knowledge base construction phase of the system, we focused on the public opinion field, extensively collecting various typical questions and corresponding answers to construct standardized question-and-answer pairs. These question-and-answer pairs are organized into independent text paragraphs in a "question + answer" fusion format, and are ultimately stored uniformly in the Elasticsearch search engine. The document loader component built into the LangChain framework has the ability to efficiently process documents of various formats. During development, by configuring the UnstructuredFileLoader's running mode to elements, the above text paragraphs can be accurately divided into individual independent text blocks, and each text block completely carries a set of question-and-answer pair information, effectively ensuring the integrity of the question-and-answer pair content.
[0100] 2. Loading of multimodal vectorized models
[0101] Multimodal data vectorization of public opinion is achieved through a pre-trained CLIP model. CLIP is a powerful open-source embedding model trained on a large text-image dataset, supporting multimodal similarity calculation and cross-modal retrieval in both Chinese and English. Within the LangChain framework, pre-trained model files can be loaded via an interface, and this embedding model can be used for cross-modal data vectorization.
[0102] 3. Vector data import
[0103] To store vector data and query keywords and semantic vectors, this system uses the Elasticsearch full-text search library to store multimodal data vectors. This paper integrates the Elasticsearch search interface into LangChain, allowing for rapid storage and retrieval of vector data through secondary development.
[0104] 4. Load the Qwen model
[0105] Qwen, an advanced large language model, has demonstrated excellent performance in multiple publicly available Chinese and English datasets, thanks to its efficient dynamic inference mechanism and memory optimization techniques. When loading the Qwen-7B large language model into the LangChain framework, a custom Qwen model class can be built using the LLM wrapper tool. To develop a custom LLM class in LangChain to adapt to the Qwen model, the development of this custom LLM class must be completed based on the input / output modules of the LangChain framework. This custom class must complete two core tasks: First, load the pre-trained model file of the customized Qwen model in the initialization method, explicitly specifying the storage path of the Qwen pre-trained file through the `model_path` parameter; second, implement the `_call` method, which must accept the `Prompt` string as an input parameter and output the corresponding `response` string as the return result.
[0106] 5. Knowledge retrieval based on vector databases
[0107] After the user-submitted question text is vectorized, the semantic similarity retrieval algorithm of the vector database can be used to retrieve the top few vectors (the number is determined by the parameter k) with the highest similarity to the question text, and return the text content corresponding to these vectors. This system uses Elasticsearch as the retrieval engine. In the LangChain framework, Elasticsearch uses cosine similarity of type dense_vector as the evaluation criterion for vector similarity by default. Within the LangChain framework, the similarity_search function of the vector database can be called to perform semantic similarity retrieval operations.
[0108] 6. Constructing prompt word templates
[0109] The core logic of using RAG technology to answer knowledge-based questions lies in leveraging a large language model to perform semantic understanding and in-depth analysis of the retrieved knowledge content, thereby locating and extracting accurate answers from the retrieved knowledge system. It's important to clarify that RAG technology does not rely on the large language model's built-in knowledge reserves; instead, it utilizes its ability to understand and analyze text semantics to accurately extract effective answers from the retrieved knowledge. To further improve the accuracy and efficiency of the large language model's answer acquisition, a well-designed Prompt template plays a crucial role. As a key information carrier directly input into the large language model, the appropriateness of the Prompt directly affects the accuracy of the model's output, which is particularly prominent in RAG application scenarios. Typically, a RAG Prompt should include three core components: a task description, background knowledge (relevant information obtained during the retrieval process), and task instructions (derived from the specific question posed by the user). These three parts work together synergistically to provide clear guidance for the large language model to generate accurate answers that meet the requirements.
[0110] 7. Answer generation
[0111] After constructing the Qwen large language model using LangChain's LLM wrapper, the model can be fused and adapted with preset prompt word templates to generate answers to user-asked questions. The specific final output results are as follows: Figure 3 and Figure 4 As shown.
[0112] This invention provides an adaptive multimodal retrieval system based on updating a public opinion knowledge base. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.
Claims
1. An adaptive multimodal retrieval system based on updating a public opinion knowledge base, characterized in that, It includes a multimodal data access module, a dynamic update module for the public opinion knowledge base, an enhanced Elasticsearch retrieval module, and a RAG retrieval enhancement generation module; The multimodal data access module is used to perform unified format conversion and feature extraction on text, image, and video public opinion data to generate structured modal feature vectors. The dynamic update module of the public opinion knowledge base is used to realize the real-time update and version management of the public opinion knowledge base through incremental crawling, semantic deduplication and timeliness filtering operations. The Elasticsearch enhanced retrieval module is used to construct a multi-dimensional vector index and perform a collaborative retrieval of vector retrieval and text retrieval based on the structured modal feature vector to obtain keyword and semantic comprehensive retrieval results; The RAG retrieval enhancement generation module is used to integrate the keyword and semantic comprehensive retrieval results with the public opinion knowledge base information to form a Prompt, and send the Prompt to the LLM large language model to generate the target answer.
2. The system as described in claim 1, characterized in that, The multimodal data access module processes public opinion data that includes two types: text, images, and videos.
3. The system as described in claim 2, characterized in that, The timeliness filtering operation is used to remove historical public opinion data from the public opinion knowledge base that exceeds the preset timeliness threshold.
4. The system as described in claim 3, characterized in that, The Elasticsearch enhanced retrieval module includes distributed data nodes and distributed machine learning embedding nodes; wherein, the distributed machine learning embedding nodes are used to generate retrieval vectors based on the structured modality feature vectors, and the distributed data nodes are used to perform collaborative retrieval operations of vector retrieval and text retrieval.
5. The system as described in claim 4, characterized in that, The distributed machine learning embedding node supports the uploading and deployment of business-tuned models or industry-customized models.
6. The system as described in claim 5, characterized in that, The RAG search enhancement generation module is used to associate and integrate the keyword and semantic comprehensive search results with the corresponding real-time updated public opinion information in the public opinion knowledge base.
7. The system as described in claim 6, characterized in that, The system is used for monitoring public opinion in government affairs.
8. The system as described in claim 7, characterized in that, The system performs the following steps: Step 1: The multimodal data access module performs unified processing on text, image, and video public opinion data, including: The Sentence-BERT pre-trained sentence encoding model is used to extract the semantic vector V of each sentence for text-based public opinion analysis. text : V test =Sentence-BERT(s i ), Among them, s i Given the input text, Sentence-BERT is a pre-trained text sentence feature extraction network; High-level semantic features V of images are extracted using the multimodal pre-trained model CLIP. img : V img =CLIP(I), Where I is the input image; After extracting keyframes from the video, image feature extraction is performed on each frame, and then the average value of the frame feature sequence is taken to obtain the video feature vector V. video : Among them, I i Let k be the i-th keyframe in the video, and k be the number of keyframes. Multimodal feature fusion generates a structured modal feature vector V. multi : V multi =αV text +βV img +γV video Where α, β, γ are modal weights, and α + β + γ = 1; Step 2, the public opinion knowledge base dynamic update module executes the following process: Step 2-1: Use web crawlers to capture text, image, and video data from publicly available public opinion platforms in real time, and access multimodal data; Step 2-2: Calculate the feature vector similarity between the newly added data and the existing data in the knowledge base, using V... new and V old cosine similarity Sim(V) new V old As a feature vector similarity: Where Sim(V) new V old )∈[0,1], representing the similarity between the old and new knowledge bases. If the similarity is greater than the preset threshold τ, it is judged as duplicate data and removed; V new and V old Representing the vectors of the old and new knowledge bases respectively, ||V new ||and||V old || represent the L2 norms of the old and new knowledge base vectors, respectively; Step 2-2: Mark each piece of public opinion data with a timestamp, periodically delete data whose timestamps exceed the preset time window, and complete the knowledge base version update; Step 3, enhance Elasticsearch retrieval; Step 4: RAG retrieval enhancement generation.
9. The system as described in claim 8, characterized in that, Step 3 includes: The Elasticsearch Enhanced Search module performs the following process based on the distributed ES cluster: Step 3-1, Vector Index Construction: Construct the structured modal feature vector V multi Write to Elasticsearch distributed data nodes to build a multi-dimensional vector index; Step 3-2, Collaborative Retrieval: After receiving the user's retrieval request, the query vector V is first generated through the Elasticsearch distributed machine learning embedding nodes. q Then, vector retrieval and text retrieval are performed simultaneously; the vector retrieval involves matching V in the vector index. q The results of the cosine similarity Top-N; the text retrieval is an ES Boolean query and OR non-full-text search performed on the user-requested keywords; Step 3-3, Result Fusion: Combine the vector retrieval and text retrieval results according to their weights to obtain the comprehensive retrieval score result based on keywords and semantics.
10. The system as described in claim 9, characterized in that, Step 4 includes: The RAG retrieval enhancement generation module performs the following procedures: Step 4-1, Information Fusion: Associate the comprehensive search results with the corresponding real-time public opinion information in the knowledge base to extract the core content; Step 4-2, Prompt assembly: Assemble the Prompt according to the format of search results, knowledge base information, and task instructions; Step 4-3: Send the Prompt to the LLM large language model to generate the final public opinion analysis answer.