Text compression strategy for knowledge base enhanced retrieval

Abstract

The invention relates to the technical field of artificial intelligence, in particular to a text compression strategy for knowledge base enhanced retrieval. The method comprises the following steps: S1, based on a pre-trained BERT answer model, identifying and extracting an input question text and an answer fragment of each retrieved related text; s2, preprocessing each segment of retrieved related text; and S3, in combination with the preprocessing result of each section of related text, performing semantic matching scoring by adopting a Reranker model, sorting the obtained scoring results of each section of related text, selecting a plurality of results according to the sorting priority, and combining the results as a combination result of the problem text. And the Reranker model evaluates the priority according to the correlation between the sentences and the questions. According to the text compression strategy for knowledge base enhanced retrieval, the pre-training question and answer model is introduced, key answer fragments in the original text are efficiently extracted to serve as core keywords, the text processing efficiency is improved, and the focusing capacity of the model on answers can be enhanced.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a text compression strategy for knowledge base enhanced retrieval. Background Technology

[0002] With the development of large-scale artificial intelligence models, knowledge base augmented retrieval (RAG) has been widely used in information retrieval and question-answering systems. RAG technology, by combining retrieval and generative models, can effectively improve the accuracy of answers to complex questions. However, in practical applications, the retrieved file blocks are often long and numerous, and even selecting the top three results may not cover the correct answer. This problem is particularly prominent when handling open-domain question answering and information-intensive scenarios.

[0003] Currently, mainstream solutions mainly focus on improving retrieval accuracy or optimizing the contextual understanding capabilities of generative models, but the following problems still exist: 1. The text compression process may lose key information. Although compression aims to simplify the input information to meet the length limit of large models, key information that is related to the question but has low similarity may be mistakenly deleted during similarity calculation and compression, which may affect the comprehensiveness and accuracy of the answer. 2. Similarity calculation depends on the performance of the semantic vector model. Text compression and ranking are highly dependent on the pre-trained semantic vector model. If the model fails to accurately capture the semantic features in a specific domain, it may lead to inaccurate similarity calculation, which will have a negative impact on information filtering and ranking. The flexibility of compression and ranking strategies is limited. The preset similarity threshold and compression strategy may lack flexibility and may not be able to be dynamically adjusted when faced with diverse user questions, resulting in inaccurate answers to some questions or poor adaptability. This leads to long text processing efficiency issues.

[0004] 3. If the text in the knowledge base changes or new strings appear, the encoding table needs to be updated, which may increase maintenance costs and requires regular updates to maintain compression efficiency; handling of unknown or new words: the method in the patent may have limitations when handling unknown or new words.

[0005] 4. If the text to be compressed contains new words or rare words not included in the encoding table, additional processing steps may be required to handle these words, which may affect compression efficiency and retrieval accuracy; a balance between compression rate and retrieval efficiency is needed.

[0006] 5. Existing mainstream text compression techniques for improving the accuracy of knowledge base retrieval have two drawbacks. First, long texts contain a lot of redundant information, and directly inputting them into the generation model will significantly increase the consumption of computing resources. Second, lengthy context may interfere with the model's ability to extract the core answer, leading to a decrease in the accuracy of the final generated result.

[0007] Therefore, effectively compressing long texts while retaining core information has become a significant challenge in knowledge base augmentation retrieval technology. Summary of the Invention

[0008] To address the aforementioned issues, the present invention aims to provide a text compression strategy for knowledge base enhanced retrieval. By introducing a pre-trained question-answering model, key answer fragments from the original text are efficiently extracted as core keywords, thereby improving text processing efficiency and enhancing the model's ability to focus on answers.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: Step S1: Based on the pre-trained BERT answer model, identify and extract the input question text and the answer fragments of each relevant text retrieved; Step S2: Preprocess each relevant text segment retrieved; Step S3: Combining the preprocessing results of each relevant text segment above, the Reranker model is used to perform semantic matching and scoring. The scoring results of each relevant text segment are sorted, and several results are selected and combined according to the sorting priority to form the combined result of the question text.

[0010] Furthermore, in step S1, the specific process of identifying and extracting key answer fragments from the input question text and each retrieved relevant text is as follows: Step S11: Match the input question text with each retrieved relevant text segment, and combine the scores of the [CLS] tags generated by the BERT answer model to determine the answer segments in each relevant text segment that match the question text; Step S12: Based on the above answer fragments, extract core keywords using TF-IDF, POS annotation, and word frequency statistics.

[0011] Furthermore, the preprocessing is specifically as follows: Step S21: Split each relevant text segment into independent sentences; Step S22: Based on the core keyword results of the answer fragment, select sentences containing the core keywords from the independent sentences of the above related text; Step S23: Based on the above filtering results, combine the filtered sentences; Step S24: Use similarity calculation to remove duplicates from the above sentence combination to obtain a new text fragment.

[0012] Furthermore, the similarity calculation specifically includes pre-similarity or Jaccard coefficient.

[0013] Furthermore, the BERT response model is the BERT-QA model.

[0014] The present invention has the following beneficial effects: 1. This invention effectively improves the accuracy and relevance of answers. By extracting key answer fragments and generating core keywords through a pre-trained BERT question-answering model, this method can quickly locate the most relevant information to the question, avoiding the drawbacks of traditional Top-N strategies that may miss key content. Furthermore, by selecting sentence combinations containing core keywords, it ensures that the model focuses on the core of the question, significantly improving the accuracy of the answers.

[0015] 2. This invention reduces computational overhead and improves system efficiency. In long text processing, the sentence combination filtering and redundancy removal strategies can significantly reduce the amount of text that needs to be evaluated, thereby significantly reducing the computational complexity of the subsequent Reranker model. Compared to directly processing the complete text, the amount of data after filtering is smaller, and the processing speed is faster, thus optimizing the entire retrieval and generation process.

[0016] 3. This invention enhances the robustness and adaptability of the system by using the Reranker model to score and rank the filtered sentence combinations. This dynamically adapts to the needs of different question types, ensuring a more reasonable priority for the output results. Even in diverse question scenarios, this strategy ensures answer quality and enhances the system's robustness.

[0017] 4. This invention improves practicality and scope of application. It is not only suitable for large-scale model-assisted question-answering scenarios, but also performs exceptionally well in enterprise knowledge management and intelligent search optimization. Through compression and rearrangement strategies, it helps enterprises quickly extract high-value content from massive amounts of documents, improving decision-making efficiency; in intelligent search, it enhances the accuracy and readability of search results. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the present invention. Detailed Implementation

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments: Step S1: Based on the pre-trained BERT answering model, identify and extract the input question text and the answer fragments of each relevant retrieved text; Step S2: Preprocess each relevant text segment retrieved; Step S3: Combining the preprocessing results of each relevant text segment, the Reranker model is used for semantic matching and scoring. The scores of each relevant text segment are sorted, and several results are combined according to the sorting priority to form the combined result of the question text. The Reranker model evaluates the priority based on the relevance of sentences to the question.

[0020] Furthermore, the results are sorted according to the scoring results (e.g., Top-1 or Top-3); the final compressed result is output for use by the subsequent RAG generation module.

[0021] Furthermore, in step S1, the specific process of identifying and extracting key answer fragments from the input question text and each retrieved relevant text is as follows: Step S11: Match the input question text with each retrieved relevant text segment, and combine the scores of the [CLS] tags generated by the BERT answer model to determine the answer segments in each relevant text segment that match the question text; Step S12: Based on the above answer fragments, extract core keywords using TF-IDF, POS annotation, and word frequency statistics.

[0022] Furthermore, the preprocessing is specifically as follows: Step S21: Split each relevant text segment into independent sentences; Step S22: Based on the core keyword results of the answer fragment, select sentences containing the core keywords from the independent sentences of the above related text; Step S23: Based on the above filtering results, combine the filtered sentences; Step S24: Use similarity calculation to remove duplicates from the above sentence combination to obtain a new text fragment.

[0023] Furthermore, the similarity calculation specifically includes pre-similarity or Jaccard coefficient.

[0024] Furthermore, the BERT answering model is the BERT-QA model, which serves as the base model, taking the input question and each piece of text retrieved.

[0025] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0026] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0027] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0028] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0029] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

Claims

1. A text compression strategy for knowledge base enhanced retrieval, characterized in that, Includes the following steps: Step S1: Based on the pre-trained BERT answering model, identify and extract the input question text and the answer fragments of each relevant retrieved text; Step S2: Preprocess each relevant text segment retrieved; Step S3: Combining the preprocessing results of each relevant text segment above, the Reranker model is used to perform semantic matching and scoring. The scoring results of each relevant text segment are sorted, and several results are selected and combined according to the sorting priority to form the combined result of the question text.

2. The text compression strategy for knowledge base enhanced retrieval according to claim 1, characterized in that, In step S1, the specific process of identifying and extracting key answer fragments from the input question text and each relevant text retrieved is as follows: Step S11: Match the input question text with each retrieved relevant text, and combine the scores of the [CLS] tags generated by the BERT answer model to determine the answer fragments in each relevant text that match the question text; Step S12: Based on the above answer fragments, extract core keywords using TF-IDF, POS annotation, and word frequency statistics.

3. The text compression strategy for knowledge base enhanced retrieval according to claim 2, characterized in that, The preprocessing is as follows: Step S21: Split each relevant text segment into independent sentences; Step S22: Based on the core keyword results of the answer fragment, select sentences containing the core keywords from the independent sentences of the above related text; Step S23: Based on the above filtering results, combine the filtered sentences; Step S24: Use similarity calculation to remove duplicates from the above sentence combination to obtain a new text fragment.

4. The text compression strategy for knowledge base enhanced retrieval according to claim 3, characterized in that, The similarity calculation specifically includes pre-similarity or Jaccard coefficient.

5. The text compression strategy for knowledge base enhanced retrieval according to claim 1, characterized in that, The BERT response model is the BERT-QA model.

Images