RAG method, system, device and medium with multi-mode and security capabilities

Abstract

The invention discloses an RAG method, system, equipment and medium with multi-mode and security capabilities, which adopt corresponding schemes: through malicious detection, unsafe access can be prevented in an early stage, and only safe and verified query can be carried out, so that the integrity and security are enhanced; multi-mode support is realized; moreover, multiple modalities (e.g., text, image, audio and video, etc.) are supported, ensuring that richer contextual answers are provided; besides, a structured and isolated entity storage scheme is used, and noise is effectively reduced by paying attention to specific entity data, so that highly accurate response with consistent context is realized. Generally speaking, the invention can improve the security, and can improve the quality and correlation of answers according to the unique characteristics of entities in the query context by focusing on the retrieval and generation of specific entity information.

Description

Technical Field

[0001] The present invention relates to the field of retrieval enhancement generation technology, and in particular to a RAG method, system, device and medium with multimodal and security capabilities. Background Art

[0002] Retrieval-Augmented Generation (RAG) is an emerging method that can significantly enhance the capabilities of LLM (Large Language Model). By leveraging external knowledge in the retrieved passages, RAG can alleviate problems such as hallucinations and inconsistencies in LLM output.

[0003] However, traditional RAG systems typically focus on document-level retrieval, which lacks the fine-grained understanding required to accurately capture entity-related details scattered across multiple sources. This limitation is further exacerbated by the mixed storage of information from different entities, resulting in retrieval noise that affects the relevance and factual accuracy of generated content.

[0004] For example, when answering questions about a specific product, a RAG system might inadvertently retrieve information about similar products, resulting in irrelevant or misleading results. In terms of multimodal data output capabilities, inaccurate data descriptions and a lack of sufficient relevant training data further exacerbate the limitations of multimodal generative models, ultimately leading to a less-than-satisfactory user experience. Furthermore, RAG systems are vulnerable to security threats such as malicious queries and document extraction attacks, which compromise data integrity and user privacy.

[0005] In view of this, the present invention is proposed. Summary of the Invention

[0006] The purpose of the present invention is to provide a RAG method, system, device and medium with multimodal and security capabilities, which can improve security and improve the quality and relevance of answers based on the unique characteristics of entities in the query context by focusing on the retrieval and generation of specific entity information.

[0007] The purpose of the present invention is achieved through the following technical solutions:

[0008] A multimodal and secure RAG method, comprising:

[0009] After performing malicious detection on the input user query q, the entities and intents are extracted and the user query is rewritten; the intent represents the answer category;

[0010] Using the extracted entities and intents, we search a pre-built entity-centric database to obtain the corresponding data subset. Entities are pre-identified from multimodal data, classified, and stored separately to form an entity-centric database.

[0011] The final answer is generated by combining the rewritten user query and the corresponding data subset.

[0012] A RAG system with multimodal and security capabilities, comprising:

[0013] An entity-centric data construction module is used to pre-identify entities from multimodal data, classify them, and store them in isolation to form an entity-centric database;

[0014] The query parser is used to detect maliciousness of the input user query q, extract entities and intents, and rewrite the user query; the intent represents the answer category;

[0015] The entity retrieval module is used to use the extracted entities and intents to search in a pre-built entity-centric database to obtain the corresponding data subset;

[0016] The answer generation module is used to generate the final answer by combining the rewritten user query and the corresponding data subset.

[0017] A processing device comprising: one or more processors; a memory for storing one or more programs;

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0019] A readable storage medium stores a computer program, which implements the aforementioned method when the computer program is executed by a processor.

[0020] It can be seen from the technical solutions provided by the present invention that: (1) through malicious detection, unsafe access can be prevented at an early stage, ensuring that only safe and verified queries can be made, thereby enhancing integrity and security; multi-modal support; (2) support for multiple modalities (for example, text, images, audio and video, etc.) to ensure that richer contextual answers are provided; (3) using a structured, isolated entity storage solution, by focusing on specific entity data, effectively reducing noise, thereby achieving highly accurate and context-consistent responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A flowchart of a RAG method with multimodality and security capabilities provided by an embodiment of the present invention;

[0023] Figure 2 A schematic diagram of a brief entity-centric question answering system provided by an embodiment of the present invention;

[0024] Figure 3 A schematic diagram of the overall framework of a RAG method with multimodality and security capabilities provided by an embodiment of the present invention;

[0025] Figure 4 A schematic diagram of a RAG system with multi-modality and security capabilities provided by an embodiment of the present invention;

[0026] Figure 5 A schematic diagram of a processing device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following is a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] First, the following terms may be used in this article:

[0029] The terms "include," "comprises," "contains," "has," or other similar expressions should be interpreted as non-exclusive. For example, "including certain technical features (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products, or manufactured articles, etc.) should be interpreted as including not only the technical features explicitly listed, but also other technical features known in the art that are not explicitly listed.

[0030] The term "consisting of" excludes any technical features not explicitly listed. If used in a claim, this term renders the claim closed, excluding any technical features other than those explicitly listed, except for conventional impurities associated with them. If this term appears only in a clause of a claim, it limits only the elements explicitly listed in that clause; elements listed in other clauses are not excluded from the claim as a whole.

[0031] The following describes in detail a multimodal and secure RAG method, system, device, and medium provided by the present invention. Any information not described in detail in the embodiments of the present invention is prior art known to those skilled in the art. For any unspecified conditions in the embodiments of the present invention, the procedures were performed in accordance with conventional conditions in the art or the conditions recommended by the manufacturer. Instruments used in the embodiments of the present invention, for which the manufacturer is not specified, are all commercially available conventional products.

[0032] Example 1

[0033] The embodiment of the present invention provides a RAG method with multi-modality and security capabilities, such as Figure 1 As shown, it mainly includes the following steps:

[0034] Step 1: After performing malicious detection on the input user query q, extract the entities and intents, and rewrite the user query.

[0035] In an embodiment of the present invention, toxicity scoring and confusion analysis are used to identify harmful content and confusing content of user query q respectively; if harmful content or confusing content is identified, the subsequent process is terminated; if harmful content and confusing content are not identified, entities are extracted through entity disambiguation technology, and the answer category (i.e., intent) required by the user is identified; at the same time, the user query is rewritten according to the set prompt words.

[0036] Step 2: Use the extracted entities and intents to search in a pre-built entity-centric database to obtain the corresponding data subset.

[0037] In an embodiment of the present invention, entities are pre-identified from multimodal data, classified, and then stored in isolation to form an entity-centric database. The preferred implementation is as follows: (1) Real multimodal data is collected. For text data, the corresponding entities are identified; for non-text data, corresponding text summary information is generated through a large language model, and then the corresponding entities are identified from it; (2) The identified entities are further divided to obtain corresponding features of the entities, and classified to obtain corresponding attribute information and the categories to which they belong; (3) The specific attribute information is stored in isolated vectorized partitions. The entities and their categories are used in an isolated manner to form an entity-centric database.

[0038] When an entity has multiple optional feature partitioning methods, the gain ratio method is used to determine the final feature selection, using f j Represents the features corresponding to the j-th partitioning method of the entity.

[0039] Calculate the gain ratio of the feature, expressed as:

[0040]

[0041] Among them, G(f j ) is the feature f j Gain ratio, IG(f j ) is the feature f j The information gain, H(f j ) is the feature f j The inner information;

[0042] Compare the gain ratios of the features corresponding to all entity division methods, and select the feature with the largest gain ratio as the feature corresponding to the entity.

[0043] In an embodiment of the present invention, the entity-centric database contains a number of entities that are classified and stored in isolation; the data of each entity includes attribute information of the entity and the category to which it belongs; the extracted entities and intentions are searched in the entity-centric database, matched with the data of each entity in the database respectively, and the data of multiple entities with the highest matching degree are selected as the corresponding entity data subset (referred to as data subset for short).

[0044] In an embodiment of the present invention, when the corresponding data subset of the extracted entity cannot be retrieved in the entity-centric database, a knowledge loss mechanism is triggered, indicating that the knowledge scope is currently exceeded and feedback is given to the user.

[0045] Step 3: Combine the rewritten user query and the corresponding data subset to generate the final answer.

[0046] In the embodiment of the present invention, the corresponding data subset is recorded as in, is the lth data subset, l = 1, 2, ..., n, n is the number of entities, corresponding to the number of data subsets; combined with the rewritten user query With data subset The most relevant k document paragraphs are retrieved by the retriever R, and finally n groups of the most relevant k document paragraphs are retrieved; the n groups of the most relevant k document paragraphs and the rewritten user query are Input into the large language model to obtain the final answer.

[0047] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method provided by the embodiment of the present invention is described in detail below with reference to specific embodiments.

[0048] 1. Overall overview of the plan.

[0049] This paper presents a framework designed to enhance entity-specific query processing and multimodal data processing. It introduces a novel entity-centric data representation that isolates information by entity to reduce noise and improve retrieval accuracy. It also integrates a unified multimodal approach supporting text, visual, and audio, and employs a proactive security strategy to apply protection measures before data access.

[0050] Current solutions suffer from confusion between similar entities. Providing precise and relevant answers is a significant challenge in various fields, such as healthcare, finance, and customer service. An entity is any unique object with identifiable attributes, such as a person, place, organization, or product, and plays a crucial role in determining the accuracy and usefulness of a response. However, traditional methods typically retrieve information across an entire text corpus, where similar text related to different entities exists, which can easily lead to information confusion and cause large language models to hallucinate. This confusion undermines the reliability of responses, necessitating more precise processing and retrieval of specific entity information.

[0051] By focusing on the retrieval and generation of specific entity information, the present invention can improve the quality and relevance of answers based on the unique characteristics of entities in the query context. Figure 2 shown.

[0052] To address this limitation, the present invention provides a solution that aims to enhance the confusion problem between similar entities in large language models. Figure 3 As shown, the solution of the present invention mainly includes four parts: an entity-centric data construction module (EDC), a query parser (QP), an entity retrieval module (ER) and an answer generation module (AG).

[0053] When using the present invention, the first step involves data preprocessing, such as Figure 3 This is shown in the lower part. This includes multi-modal processing for expression consistency, data segmentation, and isolated storage. The EDC module is responsible for these tasks.

[0054] After completing the data preprocessing phase, you can start using the question-answering function, such as Figure 2 As shown in the upper part. Given a user query q, the query parser processes it, extracts the entity e and intent i of the query q, and then rewrites the query q for use in the retrieval stage. The rewritten query is

[0055]

[0056] The entity retrieval module extracts the relevant data subset D corresponding to entity e and intent i ei The subset is extracted from the entity-centric database D, which is constructed by the EDC module:

[0057] D ei =ER(e,i,D).

[0058] The answer generation module then rewrites the query and the retrieved entity-specific data subset D ei As input, generate the final answer A:

[0059]

[0060] 2. Detailed introduction of the plan.

[0061] This section mainly introduces the above four modules respectively.

[0062] 1. Entity-centric data building blocks.

[0063] Entity-centric data building blocks organize structured data around a single entity, associating each entity with multimodal attributes, such as Figure 3 The bottom half shows this. The key to this approach lies in data isolation, which separates entity-specific information to prevent confusion and improve retrieval accuracy. By creating isolated data subsets, the system reduces interference from irrelevant information, enabling efficient and accurate retrieval and improving performance in question answering tasks.

[0064] The entity-centric data building block uses a three-stage process to process and store multimodal data.

[0065] (1) Multimodal data preprocessing.

[0066] Unlike traditional generative methods that require extensive training on non-text modalities, the present invention can process all modalities contextually, providing consistent results across text, images, audio, and video. Figure 2 Shown on the right.

[0067] The method of the present invention focuses on enhancing semantic consistency and cross-modal contextual consistency. Instead of generating data, the present invention uses existing real multimodal data and utilizes models such as WhisperGPT-4o to generate text summaries (i.e., unified text descriptions) that are consistent with the original data. This ensures that the generated summaries are consistent with the original text in terms of expression while seamlessly integrating with the existing RAG framework. This method can achieve real-time, high-precision output with minimal computational overhead.

[0068] (2) Entity recognition.

[0069] In the EDC module, a fast, cost-effective and efficient method for automatic data entity recognition is provided. Advanced keyword extraction models are used to process multimodal data. These keywords (K = {k1, k2, ..., k n ,}) are grouped into feature sets using text embedding models and cosine similarity. For example, keywords such as “refrigerator” and “washing machine” are classified as appliance features, while “kitchen” and “bathroom” are classified as usage scenario features.

[0070] Exemplarily, the keyword extraction model may use the YAKE (Yet Another Keyword Extractor) model, which is an unsupervised keyword extraction algorithm.

[0071] Of course, you can also perform manual segmentation directly based on business needs. After choosing which features to use for entity-based data segmentation, you also need to manually assign non-text multimodal data to different data subsets.

[0072] Theoretically, the feature division of each entity is consistent, but the present invention is also applicable to the selection of single entity features. When an entity has multiple optional feature division methods, in order to evaluate the features, the present invention adopts the gain ratio method. For a given set of features (F = {f1, f2, ..., f m}), where m is the number of features, corresponding to the number of feature division methods, and each feature f j The gain ratio G(f j ) is calculated as follows:

[0073]

[0074] Where IG(f j ) represents the feature f j The information gain of H(f j ) is intrinsic information. The present invention selects features with the highest gain ratio to represent entities. This process can decompose and classify large volumes of documents, ensuring maximum information gain in the entity organization process, regardless of corpus size. As a result, relevant data is appropriately categorized into structured attributes associated with each entity.

[0075] (3) Secure isolated storage.

[0076] To process data, the present invention first divides the data into entities and stores them in isolated, vectorized partitions. By separating the data in this way, the present invention can implement precise access control and manage permissions for specific entities, significantly reducing the exposure of sensitive information. This structure not only strengthens security but also improves retrieval accuracy because each query only accesses the relevant subset of data, reducing the risk of unauthorized access.

[0077] like Figure 3 As shown in the figure, attributes are specific information of entities, such as text, images, videos, etc. Intents and entities are both labels. Intent labels are the names of specific feature division methods. For example, secondary classification 1 and secondary classification 2 are both names corresponding to specific feature division methods.

[0078] After data preprocessing, the present invention only uses tags associated with entities, without requiring direct access to detailed document content. This setup enables the present invention's security mechanisms to be fully operational before any document is accessed. The present inventors refer to this strategy as pre-emptive security design, which provides strong protection against a range of attack vectors, such as document extraction and hallucination attacks.

[0079] 2. Query parser.

[0080] The query parser takes user queries as input and outputs entities, intents, and rewritten queries. The main process is as follows:

[0081] Step 1: Check for malicious content or unsafe instructions. If detected, reject and explain; otherwise, proceed as follows.

[0082] (1.1) Determine the entity the user is currently discussing, referencing previously mentioned entities when necessary.

[0083] (1.2) Organize the user's current input into a more concise statement.

[0084] (1.3) Infer the user's intent based on what the user wants to know.

[0085] Step 2: Output.

[0086] (2.1) Indicators of malicious content, including toxicity and obfuscation.

[0087] (2.2) Signs indicating the existence of entities and intentions.

[0088] (2.3) The entity that the user is currently discussing can be selected from a predefined list.

[0089] (2.4) Intent selected from a predefined list (based on the secondary classification of data feature entities).

[0090] (2.5) Rewritten user query.

[0091] (2.6) Reasons for rejecting or accepting an answer, for example, if malicious content was identified, the answer was rejected; if no malicious content was identified, the answer was accepted.

[0092] (1) Malicious identification.

[0093] like Figure 3 As shown, the Query Parser (QP) includes a malicious query detection component that preemptively scans user input for harmful or obfuscated content using toxicity scoring and obfuscation analysis. Queries that exceed toxicity or obfuscation thresholds are flagged and blocked from further processing, preventing unsafe access at an early stage. This filtering ensures that only safe, validated queries are processed, enhancing system integrity and security.

[0094] (2) Extract entities and intents.

[0095] In complex conversational contexts, user queries may contain incomplete or ambiguous entity information. The query parser module addresses this issue by using advanced entity disambiguation techniques to refine queries based on contextual clues and conversation history.

[0096] This process ensures that only the most relevant entities are selected for retrieval, without prematurely abandoning ambiguous queries. In addition, the module identifies the category of answers desired by the user as intent.

[0097] (3)Query rewriting.

[0098] The query parser module rewrites the original query into a more concise and professional form while retaining its essential meaning. This rewriting process removes noise and irrelevant information, ensuring that the query is well-structured and focused on the core information required.

[0099] 3. Entity retrieval module.

[0100] (1) Data subset matching.

[0101] like Figure 3 As shown, by matching user-identified entities and intents with specific subsets of data, it accurately locates relevant information, maintaining high accuracy while reducing processing. For multiple entities, it retrieves data for each entity simultaneously, ensuring accurate presentation without interfering with each other. This entity-centric approach avoids the information confusion problem common in traditional systems that use unseparated data, where the lack of entity isolation leads to mixed and misleading output.

[0102] (2) Beyond the scope of the knowledge base.

[0103] The out-of-scope knowledge base mechanism is activated when a query contains one or more unrecognized entities. For each entity in the query, the system verifies whether it exists in the knowledge base. If any entity does not exist, the query is considered out-of-scope and the knowledge gap mechanism is triggered.

[0104] Once the out-of-knowledge mechanism is triggered, feedback is provided to the user indicating which entities are out of the scope of the database (knowledge base). This enables users to understand the knowledge boundaries of the system and adjust their queries accordingly.

[0105] 4. Answer generation module.

[0106] like Figure 3 As shown in

[15] , the present invention can be seamlessly integrated with the state-of-the-art RAG framework, which typically consists of an LLM (M), a dataset (D), and a retriever (R). In a standard RAG setting, given a user query (q), the system retrieves the k most relevant document paragraphs from the dataset (D) using a retriever (R), and then uses the LLM to generate an answer A (this process is referred to as the answer generation module AG):

[0107]

[0108] A=AG(q,R(q,D))

[0109] In the embodiment of the present invention, the search engine is adjusted and the user query rewritten by the query parser is The original query q is replaced by . Also, the retriever does not use the entire dataset D, but a subset of the data obtained from the entity retrieval module Each entity in the data is retrieved independently from a specific subset of data. Each subset is processed separately to ensure that the most relevant information is collected for each entity:

[0110]

[0111] Among them, the symbol e represents entity, i represents intention, is the selected lth data subset, Represents a subset of data The k most relevant document paragraphs retrieved from .

[0112] Once the independent searches are complete, the answer generation module merges the searches for all entity subsets to produce a single, cohesive answer A:

[0113]

[0114] This integration enables the RAG framework to leverage entity-centric information while maintaining its efficiency. By processing each entity subset independently during the retrieval phase and then aggregating the results, the answer generation module provides a unified response that accurately reflects the information relevant to all entities in the query.

[0115] The above solution provided by the embodiment of the present invention mainly has the following advantages:

[0116] (1) Entity Storage Accuracy. The present invention adopts a structured and isolated entity storage system to effectively reduce noise by focusing on specific entity data, thereby achieving highly accurate and context-consistent responses.

[0117] (2) Enhanced security. The present invention adopts a pre-positioned security strategy with malicious identification and knowledge loss detection functions, which can reduce illusions and ensure system integrity.

[0118] (3) Multimodal support. Compared with traditional text-only systems, the present invention allows the use of multiple data types such as text, images, audio, and video, ensuring richer contextual answers.

[0119] 3. Experimental description.

[0120] In order to verify the superiority of the present invention, the method of the present invention is compared with several state-of-the-art RAG systems as references, including basic RAG, DSP, Self-RAG, ReAct and Self-Ask, and the above five schemes are used as baseline schemes.

[0121] 1. Test dataset construction.

[0122] To evaluate our approach, we conducted experiments using the latest domain-specific car brand data publicly available on the internet. We ensured that the dataset we used was carefully curated to exclude any content typically found in LLM training corpora. With the assistance of the GPT-4o large language model, the data was meticulously cleaned and rigorously removed of personally identifiable information and any content deemed offensive. The resulting dataset contained 274 different car brands and a total of 50,665 associated attributes.

[0123] In the experiment, an evaluation dataset consisting of 2,658 pairs of questions and answers was constructed. These questions and answers were all collected from the internet, including 2,400 text questions and 268 non-text questions to ensure the accuracy of multimodal (text, image, audio) answers. In addition, 200 malicious questions were generated using GPT-4o to test attack detection capabilities, 200 questions were generated to test document extraction attack detection capabilities, and 200 irrelevant questions were manually selected to evaluate robustness against hallucination attacks.

[0124] 2. Quantitative evaluation.

[0125] In the experiment, the performance of the five baseline methods mentioned above was compared with our method. Since the baseline solutions do not support multimodal data, only 2,400 text-based question-answer pairs were used in the experiment to ensure a fair comparison, as shown in Table 1.

[0126] Table 1: Comparison of the accuracy of each method with and without the use of the present invention

[0127]

[0128]

[0129] Here, using the present invention means integrating the corresponding baseline solution with the present invention.

[0130] After using the present invention, the performance of all baseline schemes has been continuously improved. The direct method implemented using basic RAG achieved the most significant improvement, and the accuracy increased by 0.25 after combining with the present invention. Self-RAG and ReAct also showed significant improvements, with the accuracy increased by 0.14 respectively. DSP and Self-Ask increased the accuracy by 0.12 and 0.13 respectively when used in combination with the present invention. These results highlight the effectiveness of the present framework in enhancing the handling of confusion between similar entity tasks in various RAG-based methods. The excellent performance of the present invention can be attributed to its entity-centric data building module, EDC, which enables more accurate retrieval and minimizes the noise introduced by mixed entity information.

[0131] (1) Recall of entity retrieval.

[0132] In the experiment, the recall rates of the complete retrieval method and the entity retrieval method were evaluated in the top 1 and top 5 categories, as shown in Table 2.

[0133] Table 2: Comparison of recall rates between full data retrieval and entity-based retrieval

[0134] method Recall @1 Recall @5 Full data retrieval 0.3 0.67 Entity-based retrieval 0.97(+0.58) 0.98(+0.31)

[0135] The recall performance of the two retrieval methods differed significantly. For full-text retrieval, Recall@1 was 0.39 and Recall@5 was 0.67. In contrast, entity retrieval achieved much higher recall. Entity matching achieved a Recall@1 of 0.97, an improvement of 0.58, and Recall@5 of 0.98, an improvement of 0.31. Compared to full document retrieval, entity retrieval demonstrated the potential of this focused approach to improve the relevance of top-ranked results.

[0136] (2) Multimodality and attack detection statistics.

[0137] In the experiment, the performance of the present invention in identifying user intent (determining which data modality to select) and detecting various types of attacks was evaluated, as shown in Table 3.

[0138] Table 3: Intent recognition and attack defense accuracy statistics

[0139] type Number of errors / total Accuracy Intent Recognition 80 / 2658 0.97 Non-text Q&A 43 / 258 0.83 Malicious Detection 4 / 200 0.98 Document extraction attacks 3 / 200 0.98 Hallucination Attack 5 / 200 0.98

[0140] The present invention achieved 97% accuracy in intent recognition, with only 80 errors out of 2,658 queries. In a test of 268 non-text responses, the present invention achieved 83% accuracy, comparable to the accuracy of text-based responses. Its accuracy reached 98% in detecting malicious attacks, document extraction, and hallucination attacks, demonstrating its robust ability to address a wide range of threats and ensure response integrity.

[0141] 3. Qualitative assessment.

[0142] In a qualitative analysis, the experiment compared the accuracy, comprehensiveness, and safety of the DSP method with and without the present invention, as shown in Table 4. For fact-based questions (e.g., What is the wheelbase of the Fox Alpha S?), the present invention provided the correct measurement (2915 mm), while the baseline model gave an incorrect value (2980 mm), demonstrating improved accuracy. For descriptive questions (e.g., What colors and interior options are available for the Fengguang MINI?), the present invention provided more detailed answers, listing specific colors such as Sprite Green and Mist Rose, highlighting its superior comprehensiveness.

[0143] Table 4: Example of DSP dialogue with and without the present invention

[0144]

[0145] In security-focused tests, the method using our invention consistently outperformed the baseline. For malicious attack problems (such as car GPS location requests), the method not only refused to provide information but also clearly explained the privacy and security risks involved. In document extraction attacks, the baseline model provided full access to the document, while the method using our invention rejected the request and emphasized security. For hallucination attack problems, the method using our invention provided accurate responses, while the baseline model generated hallucinated content. These results show that the method's ability to handle sensitive information and prevent security vulnerabilities is enhanced, and performance across all indicators is significantly improved in resolving confusion between similar entities.

[0146] The experiments also evaluated the effectiveness of the latest LLM and the use of our method. The results in Table 5 show that the use of our method consistently provides accurate answers, while GPT-4o, OpenAIo1, and Claude, when used alone, can produce inaccurate or hallucinatory results, highlighting the robustness and versatility of our method.

[0147] Table 5: Comparison results between the present invention and the latest large language model

[0148]

[0149]

[0150] Through the description of the above embodiments, those skilled in the art will clearly understand that the above embodiments can be implemented through software or by using software plus a necessary general-purpose hardware platform. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) and includes a number of instructions for causing a computer device (such as a personal computer, a server, or a network device) to execute the methods described in the various embodiments of the present invention.

[0151] Example 2

[0152] The present invention also provides a RAG system with multi-modality and security capabilities, which is mainly used to implement the method provided in the above embodiment, such as Figure 4 As shown, the system mainly includes:

[0153] An entity-centric data construction module is used to pre-identify entities from multimodal data, classify them, and store them in isolation to form an entity-centric database;

[0154] The query parser is used to detect maliciousness of the input user query q, extract entities and intents, and rewrite the user query; the intent represents the answer category;

[0155] The entity retrieval module is used to use the extracted entities and intents to search in a pre-built entity-centric database to obtain the corresponding data subset;

[0156] The answer generation module is used to generate the final answer by combining the rewritten user query and the corresponding data subset.

[0157] Considering that the main technical details involved in the system have been introduced in detail in the previous embodiments, they will not be repeated here.

[0158] Those skilled in the art will clearly understand that for the convenience and brevity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0159] Example 3

[0160] The present invention also provides a processing device, such as Figure 5 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided by the aforementioned embodiment.

[0161] Furthermore, the processing device further includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0162] In the embodiment of the present invention, the specific types of the memory, input device, and output device are not limited; for example:

[0163] The input device can be a touch screen, image acquisition device, physical button or mouse;

[0164] The output device may be a display terminal;

[0165] The memory may be a random access memory (RAM) or a non-volatile memory, such as a disk memory.

[0166] Example 4

[0167] The present invention also provides a readable storage medium storing a computer program, which implements the method provided in the above embodiment when the computer program is executed by a processor.

[0168] In the embodiments of the present invention, the computer-readable storage medium may be provided in the aforementioned processing device, for example, as a memory in the processing device. Alternatively, the computer-readable storage medium may be a USB flash drive, a removable hard drive, a read-only memory (ROM), a magnetic disk, or an optical disk, among other media capable of storing program code.

[0169] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or any form of implication that the information constitutes prior art already known to those skilled in the art.

Claims

1. A RAG method with multimodality and security capabilities, characterized in that: include: After performing malicious detection on the input user query q, the entities and intents are extracted and the user query is rewritten; the intent represents the answer category; Using the extracted entities and intents, we search a pre-built entity-centric database to obtain the corresponding data subset. Entities are pre-identified from multimodal data, classified, and stored separately to form an entity-centric database. The final answer is generated by combining the rewritten user query and the corresponding data subset.

2. A RAG method with multimodality and security capabilities according to claim 1, characterized in that: After performing malicious detection on the input user query q, extracting entities and intents, and rewriting the user query include: Use toxicity scoring and obfuscation analysis to identify harmful content and obfuscated content in user query q. If harmful content or obfuscated content is identified, the subsequent process is terminated. If no harmful or confusing content is identified, the entity disambiguation technology is used to extract the entity and identify the type of answer the user requires; at the same time, the user query is rewritten according to the set prompt words.

3. The RAG method with multimodality and security capabilities according to claim 1, characterized in that: The extracted entities and intents are used to search a pre-built entity-centric database to obtain corresponding data subsets including: The entity-centric database includes a number of entities that are classified and stored separately; the data of each entity includes attribute information of the entity and the category to which it belongs; The extracted entities and intentions are searched in an entity-centric database, matched with the data of each entity in the database, and the data of multiple entities with the highest matching degree are selected as the corresponding data subset.

4. The RAG method with multimodality and security capabilities according to claim 1 or 3, characterized in that: Also includes: When the extracted entity cannot retrieve the corresponding data subset in the entity-centric database, the knowledge loss mechanism is triggered, indicating that the current knowledge is beyond the scope and feedback is given to the user.

5. The RAG method with multimodality and security capabilities according to claim 1 or 3, characterized in that: The step of pre-identifying entities from multimodal data, classifying them, and storing them in isolation to form an entity-centric database includes: Collect real multimodal data. For text data, identify the corresponding entities. For non-text data, generate the corresponding text summary information through a large language model, and then identify the corresponding entities from it. The identified entities are further divided to obtain the corresponding features of the entities, and then classified to obtain the corresponding attribute information and the category to which they belong; Specific attribute information is stored in isolated vectorized partitions, and the isolation method uses entities and their categories to form an entity-centric database.

6. The RAG method with multimodality and security capabilities according to claim 5, characterized in that: Ways to obtain features corresponding to entities include: When an entity has multiple feature division methods, the gain ratio method is used to determine the corresponding features, using f j Represents the features corresponding to the j-th partitioning method of the entity; Calculate the gain ratio of the feature, expressed as: Among them, G(f j ) is the feature f j Gain ratio, IG(f j ) is the feature f j The information gain, H(f j ) is the feature f j The inner information; Compare the gain ratios of the features corresponding to all entity division methods, and select the feature with the largest gain ratio as the feature corresponding to the entity.

7. The RAG method with multimodality and security capabilities according to claim 1, characterized in that: Generating a final answer by combining the rewritten user query and the corresponding data subset includes: The corresponding data subset is denoted as in, is the lth data subset, l = 1, 2, ..., n, n is the number of entities, corresponding to the number of data subsets; Combined with rewritten user queries With data subset Retrieve the most relevant k document paragraphs through the retriever R, and finally retrieve n groups of the most relevant k document paragraphs; The n groups of k most relevant document paragraphs and the rewritten user query Input into the large language model to obtain the final answer.

8. A RAG system with multimodal and security capabilities, characterized in that: include: An entity-centric data construction module is used to pre-identify entities from multimodal data, classify them, and store them in isolation to form an entity-centric database; The query parser is used to detect maliciousness of the input user query q, extract entities and intents, and rewrite the user query; the intent represents the answer category; The entity retrieval module is used to use the extracted entities and intents to search in a pre-built entity-centric database to obtain the corresponding data subset; The answer generation module is used to generate the final answer by combining the rewritten user query and the corresponding data subset.

9. A processing device, characterized in that include: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images