Information retrieval method and system based on RAG and large language model

Abstract

The invention discloses an information retrieval method and system based on an RAG and a large language model, and belongs to the technical field of intelligent office. The method comprises the following steps: analyzing different types of documents by adopting a long and short mixed granularity method and a preset splitting strategy to obtain text fragments of long and short granularities aiming at the different types of documents, and constructing a long and short mixed granularity knowledge base on the basis of the text fragments of the long and short granularities; when a user question is received, adjusting the user question into a specific query statement by using a multi-round query rewriting mechanism based on a large language model; and taking the specific query statement as a retrieval statement, performing retrieval in the long and short mixed granularity knowledge base by using a retrieval enhancement generation technology (RAG) to obtain retrieval results, sorting the retrieval results according to release dates of knowledge in the retrieval results, and preferentially displaying the retrieval results sorted in the front. The application of the method improves the accuracy of question and answer retrieval query.

Description

Technical Field

[0001] The present invention relates to the field of intelligent office technology, and more specifically, to an information retrieval method and system based on RAG and a large language model. Background Art

[0002] In the field of information retrieval, especially for applications in smart office scenarios, significant progress has been made in recent years, mainly due to the development of large language models (LLMs) and retrieval-augmented generation (RAG) technology. RAG technology effectively improves the performance and accuracy of question-answering systems by combining traditional information retrieval methods with large language models. Such systems usually first retrieve text fragments related to the question from a large number of documents, and then input these fragments as additional context into a large language model to generate more accurate answers.

[0003] Early RAG systems mainly relied on a single retrieval mechanism, such as BM25 or vector retrieval, which performed well in processing simple queries, but their performance was often limited when faced with complex queries or when searching across multi-source heterogeneous data. To overcome this limitation, subsequent research began to explore multi-source retrieval and methods that integrate different retrieval strategies to improve recall and accuracy. For example, combining keyword retrieval, vector retrieval and other methods can give full play to their respective advantages in different scenarios, thereby improving overall performance.

[0004] Query rewriting technology is also one of the important means to improve information retrieval results. By rewriting user queries, ambiguity in queries can be reduced, making them more accurately reflect users' actual information needs. Traditional query rewriting methods are mainly based on manual rules or statistical machine learning methods, while in recent years, query rewriting methods based on large models have shown stronger potential and effects. By learning a large number of query-rewrite pairs, this type of method can automatically generate higher-quality query rewrites, thereby improving retrieval accuracy.

[0005] In smart office scenarios, in addition to accuracy and completeness, timeliness is also one of the key factors that information retrieval systems need to consider. Since documents in corporate knowledge bases may be updated frequently, how to ensure that the retrieval system can prioritize returning the latest relevant information is a challenging problem. Some studies have attempted to solve this problem through time-aware sorting strategies. For example, time weights are introduced into the sorting function so that documents with more recent updates are ranked higher in the sorting results. Summary of the invention

[0006] In order to give full play to the advantages of large language model and information retrieval, the present invention proposes an information retrieval method based on RAG and large language model, including:

[0007] The long-short mixed granularity method and the preset stripping strategy are adopted to parse different types of documents to obtain text segments of long and short granularities for the different types of documents, and a long-short mixed granularity knowledge base is constructed based on the text segments of long and short granularities;

[0008] When receiving a user question, a multi-round query rewriting mechanism is used based on a large language model to adjust the user question into a specific query statement;

[0009] The specific query statement is used as a search statement, and the search enhancement generation technology RAG is used to search in the long-short mixed granularity knowledge base to obtain search results, and the search results are sorted according to the release date of the knowledge in the search results, and the search results with the highest ranking are displayed first.

[0010] Optionally, a long-short mixed granularity method and a preset stripping strategy are used to parse different types of documents, including:

[0011] Converting all types of documents into target document data having a hierarchical information representation form;

[0012] The target document data is preliminarily segmented to divide the target document data into a plurality of short-granularity text segments, and incomplete segments in the plurality of short-granularity segments are merged into a plurality of long-granularity text segments.

[0013] Optionally, different types of documents, including at least one of the following:

[0014] PDF documents, Word documents and Excel documents.

[0015] Optional, hierarchical information representation of the target document data, including:

[0016] The first level is the title, the second level is the sub-title and the third level is the paragraph.

[0017] Optionally, performing preliminary segmentation on the target document data to divide the target document data into a plurality of short-grained text segments, and merging incomplete segments in the plurality of short segments into a plurality of long-grained text segments, includes:

[0018] Presetting a maximum length threshold of a chunk, determining the chapter structure of the target document data, and dividing the target document data into a plurality of short-grained text segments according to the maximum length threshold of the chunk based on the chapter structure;

[0019] If the level of the next text segment among the multiple short-grained text segments is lower than the level of the current text segment, and the next text segment does not exceed the maximum length threshold of a chunk after being merged with the current text segment, then the next text segment and the current text segment are merged into a short-grained text segment;

[0020] The completeness of each of the multiple short-grained text fragments is determined, incomplete text fragments from the multiple short-grained text fragments are screened out, and the incomplete fragments are merged into multiple long-grained text fragments.

[0021] Optionally, each piece of data in the long-short mixed granularity knowledge base includes:

[0022] At least one short-grained text segment and at least one long-grained text segment.

[0023] Optionally, the method further includes:

[0024] Acquire the query dialogue information of the user, train the original large language model based on the query dialogue information of the user, adjust the model parameters of the original large language model, and generate a large language model;

[0025] The user's query dialogue information includes: the user's original query information, context information of the original query information, and the user's expected information.

[0026] On the other hand, the present invention also proposes an information retrieval system based on RAG and a large language model, comprising:

[0027] The knowledge base unit is used to parse different types of documents by adopting a long-short mixed granularity method and a preset stripping strategy to obtain text segments of long and short granularities for the different types of documents, and to construct a long-short mixed granularity knowledge base based on the long and short text segments;

[0028] A query unit, configured to, when receiving a user question, adjust the user question into a specific query statement using a multi-round query rewriting mechanism based on a large language model;

[0029] The retrieval unit is used to use the specific query statement as the retrieval statement, use the retrieval enhancement generation technology RAG to search in the long-short mixed granularity knowledge base, obtain the retrieval results, and sort the retrieval results according to the release date of the knowledge in the retrieval results, giving priority to displaying the retrieval results with the highest ranking.

[0030] Optionally, the knowledge base unit uses a long-short mixed granularity method and a preset stripping strategy to parse different types of documents, including:

[0031] Converting all types of documents into target document data having a hierarchical information representation form;

[0032] The target document data is preliminarily segmented to divide the target document data into a plurality of short-granularity text segments, and incomplete segments in the plurality of short-granularity segments are merged into a plurality of long-granularity text segments.

[0033] Optionally, different types of documents, including at least one of the following:

[0034] PDF documents, Word documents and Excel documents.

[0035] Optional, hierarchical information representation of the target document data, including:

[0036] The first level is the title, the second level is the sub-title and the third level is the paragraph.

[0037] Optionally, the knowledge base unit performs preliminary segmentation on the target document data, divides the target document data into a plurality of short-grained text segments, and merges incomplete segments in the plurality of short segments into a plurality of long-grained text segments, including:

[0038] Presetting a maximum length threshold of a chunk, determining the chapter structure of the target document data, and dividing the target document data into a plurality of short-grained text segments according to the maximum length threshold of the chunk based on the chapter structure;

[0039] If the level of the next text segment among the multiple short-grained text segments is lower than the level of the current text segment, and the next text segment does not exceed the maximum length threshold of a chunk after being merged with the current text segment, then the next text segment and the current text segment are merged into a short-grained text segment;

[0040] The completeness of each of the multiple short-grained text fragments is determined, incomplete text fragments from the multiple short-grained text fragments are screened out, and the incomplete fragments are merged into multiple long-grained text fragments.

[0041] Optionally, each piece of data in the long-short mixed granularity knowledge base includes:

[0042] At least one short-grained text segment and at least one long-grained text segment.

[0043] Optionally, the query unit is further configured to:

[0044] Acquire the query dialogue information of the user, train the original large language model based on the query dialogue information of the user, adjust the model parameters of the original large language model, and generate a large language model;

[0045] The user's query dialogue information includes: the user's original query information, context information of the original query information, and the user's expected information.

[0046] In yet another aspect, the present invention further provides a computing device, comprising: one or more processors;

[0047] a processor for executing one or more programs;

[0048] When the one or more programs are executed by the one or more processors, the above-described method is implemented.

[0049] In yet another aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed, the method described above is implemented.

[0050] Compared with the prior art, the present invention has the following beneficial effects:

[0051] The present invention provides an information retrieval method based on RAG and a large language model, including: using a long-short mixed granularity method and a preset stripping strategy to parse different types of documents to obtain long and short text fragments of two granularities for the different types of documents, and constructing a long-short mixed granularity knowledge base based on the long and short text fragments; when receiving a user question, using a multi-round query rewriting mechanism based on a large language model to adjust the user question to a specific query statement; using the specific query statement as a retrieval statement, using the retrieval enhancement generation technology RAG to search in the long-short mixed granularity knowledge base to obtain retrieval results, and sorting the retrieval results according to the release date of the knowledge in the retrieval results, and giving priority to displaying the top-ranked retrieval results. The application of the present invention improves the accuracy of question-answer retrieval queries. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 is a flow chart of the method of the present invention;

[0053] Figure 2 It is a principle diagram of an embodiment of the method of the present invention;

[0054] Figure 3 is a flow chart of an embodiment of the method of the present invention;

[0055] Figure 4 It is a structural diagram of the system of the present invention. DETAILED DESCRIPTION

[0056] Now, exemplary embodiments of the present invention are described with reference to the accompanying drawings. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. These embodiments are provided to disclose the present invention in detail and completely and to fully convey the scope of the present invention to those skilled in the art. The terms used in the exemplary embodiments shown in the accompanying drawings are not intended to limit the present invention. In the accompanying drawings, the same units / elements are marked with the same reference numerals.

[0057] Unless otherwise specified, the terms (including technical terms) used herein have the commonly understood meanings to those skilled in the art. In addition, it is understood that the terms defined in commonly used dictionaries should be understood to have the same meanings as those in the context of the relevant fields, and should not be understood as idealized or overly formal meanings.

[0058] Embodiment 1:

[0059] The present invention proposes an information retrieval method based on RAG and large language model, such as Figure 1 As shown, including:

[0060] Step 1: using a long-short mixed granularity method and a preset stripping strategy to parse different types of documents to obtain text segments of long and short granularities for the different types of documents, and constructing a long-short mixed granularity knowledge base based on the text segments of long and short granularities;

[0061] Step 2: When receiving a user question, a multi-round query rewriting mechanism is used based on a large language model to adjust the user question into a specific query statement;

[0062] Step 3: Use the specific query statement as the search statement, use the search enhancement generation technology RAG to search in the long-short mixed granularity knowledge base, obtain the search results, and sort the search results according to the release date of the knowledge in the search results, giving priority to displaying the search results with higher rankings.

[0063] Among them, the long-short mixed granularity method and the preset stripping strategy are used to parse different types of documents, including:

[0064] Converting all types of documents into target document data having a hierarchical information representation form;

[0065] The target document data is preliminarily segmented to divide the target document data into a plurality of short-granularity text segments, and incomplete segments in the plurality of short-granularity segments are merged into a plurality of long-granularity text segments.

[0066] The different types of documents include at least one of the following:

[0067] PDF documents, Word documents and Excel documents.

[0068] The hierarchical information representation of the target document data includes:

[0069] The first level is the title, the second level is the sub-title and the third level is the paragraph.

[0070] The target document data is initially segmented into a plurality of short-grained text segments, and incomplete segments in the plurality of short segments are merged into a plurality of long-grained text segments, including:

[0071] Presetting a maximum length threshold of a chunk, determining the chapter structure of the target document data, and dividing the target document data into a plurality of short-grained text segments according to the maximum length threshold of the chunk based on the chapter structure;

[0072] If the level of the next text segment among the multiple short-grained text segments is lower than the level of the current text segment, and the next text segment does not exceed the maximum length threshold of a chunk after being merged with the current text segment, then the next text segment and the current text segment are merged into a short-grained text segment;

[0073] The completeness of each of the multiple short-grained text fragments is determined, incomplete text fragments from the multiple short-grained text fragments are screened out, and the incomplete fragments are merged into multiple long-grained text fragments.

[0074] Among them, each piece of data in the long-short mixed granularity knowledge base includes:

[0075] At least one short-grained text segment and at least one long-grained text segment.

[0076] The method further includes:

[0077] Acquire the query dialogue information of the user, train the original large language model based on the query dialogue information of the user, adjust the model parameters of the original large language model, and generate a large language model;

[0078] The user's query dialogue information includes: the user's original query information, context information of the original query information, and the user's expected information.

[0079] Among them, the RAG technology and information retrieval principle used in the present invention are as follows Figure 2 and Figure 3 As shown, in the present invention, it mainly includes:

[0080] The method (strategy) for building and splitting documents with mixed long and short granularity is as follows:

[0081] In the information retrieval task of smart office scenarios, the document database construction and stripping strategy is crucial to improve the accuracy and efficiency of retrieval. Traditional document processing methods often use a single granularity, such as full text or fixed-size paragraphs, which is inadequate when dealing with complex queries or when specific information fragments need to be accurately recalled. Therefore, we introduced a document database construction and stripping strategy with mixed long and short granularity, aiming to ensure that the integrity of the text fragments fed into the large model is maintained while recalling the text fragments most relevant to the question.

[0082] Specifically, they include the following:

[0083] First, different types of documents are parsed uniformly. This process covers content extraction from documents in various formats (such as PDF, Word, Excel, etc.) and preserves the format and hierarchical information of the original document to the greatest extent possible. To achieve this goal, we convert the document into a data representation with hierarchical information based on heuristic rules for various title types. This data structure not only records the text content of the document, but also clarifies the hierarchical relationship between the various parts, such as title, subtitle, paragraph, etc.

[0084] Based on this data form with hierarchical information, the document is preliminarily segmented. The specific method is to first set a maximum length threshold of a chunk. The threshold in this patent is set to 438, and then merge operations are performed based on the chapter structure of the full text. If the level of the next segment is lower than the current segment, and the length of the two after merging does not exceed the threshold, the merge operation is performed; otherwise, if the hierarchical structure of the next segment is higher than the current segment, it indicates that the topic has changed, and merge operations are avoided as much as possible. This method realizes the effective merging of short segments used for retrieval, thereby improving the efficiency and accuracy of retrieval.

[0085] However, shorter fragments may cause incomplete content when the large model performs reading comprehension. For example, when the user's question is "Which units and individuals did the company commend on May 4, 2024?", the most relevant fragment may only contain part of the answer due to length restrictions, thus affecting the completeness of the answer generated by the large model. To solve this problem, we further merged the above 438-character small chunks into 3,000-character fragments. This not only maintains retrieval efficiency, but also ensures that the large model can obtain sufficiently complete content during the reading comprehension process, thereby generating more accurate answers.

[0086] Finally, the text fragments of 438 characters and 3000 characters are unified into a database. Each piece of data contains fragments of both long and short granularity. The short-granularity fragments are used to build semantic indexes and keyword indexes to quickly retrieve text fragments related to user queries; while the long fragments are used as reference fragments when they are subsequently sent to the large model to ensure that the large model can generate accurate answers based on complete information. This mixed-granularity document database building and stripping strategy aims to improve the accuracy and completeness of information retrieval, thereby enhancing user experience.

[0087] The multi-round query rewriting mechanism is as follows:

[0088] In the process of information retrieval, the expression of user queries is often ambiguous or does not fully conform to the expression of the system's internal documents, which will lead to inaccurate or incomplete retrieval results. In order to solve this problem, a multi-round query rewriting mechanism is proposed to iteratively optimize user queries, reduce ambiguity, and improve the accuracy of retrieval targets. This mechanism can not only rewrite user queries in multiple rounds, but also rewrite queries in a time-sensitive manner according to the user's timeliness requirements to ensure the timeliness of retrieval results.

[0089] Specifically, they include the following:

[0090] Multiple rounds of rewriting model fine-tuning: First, fine-tune the large language model used for query rewriting to improve its rewriting capabilities in context-related scenarios. During the fine-tuning process, the model is trained using a large number of samples containing the user's original query, context information, and the desired rewritten query. In this way, the model can learn how to rewrite the user's query into a more precise and specific query statement while taking into account the context information.

[0091] Multiple rounds of rewriting in user interaction: During the interaction between users and information retrieval systems, users' queries are often adjusted based on the feedback from the previous round of retrieval results in the hope of obtaining more accurate information. The multi-round query rewriting mechanism can capture these changes and rewrite each round of user queries. During the rewriting process, the mechanism not only considers the current round of queries, but also refers to previous queries and retrieval results to ensure the continuity and consistency of the rewriting.

[0092] Time-sensitive query rewriting: In smart office scenarios, the timeliness of information is a very important consideration. Users may want to retrieve recently updated documents or information. Therefore, the query rewriting mechanism also has time-sensitive capabilities. When the user query contains clear time requirements, the mechanism automatically incorporates absolute time information into the rewritten query to ensure that the retrieved fragments meet the timeliness required by the user. For example, if the user queries "Which units and individuals were commended on May 4th this year", the rewriting mechanism will ensure that the retrieved documents are updated during or after May 4th, 2024.

[0093] The timeliness is sorted as follows:

[0094] At the same time, the retrieval ranking architecture has been significantly upgraded, especially in terms of timeliness optimization. The previous ranking method failed to fully consider the timeliness of documents, resulting in outdated or irrelevant results being ranked higher. To address this problem, we developed a new timeliness-based ranking strategy that considers the publication date of the document and adjusts its ranking accordingly. This upgrade ensures that newer and more relevant documents are displayed first in the search results, significantly improving the overall quality and practicality of the retrieval system.

[0095] The present invention introduces a document database building and stripping strategy with mixed long and short granularity, which can maintain the integrity of the text fragments fed into the large model;

[0096] The present invention introduces a multi-round query rewriting mechanism, which reduces sentence ambiguity and improves the accuracy of retrieval targets by iteratively optimizing user queries;

[0097] In view of the characteristics of office scenarios, the present invention incorporates timeliness into the document sorting strategy, solving the problem of large models referencing outdated fragments.

[0098] Embodiment 2:

[0099] The present invention also proposes an information retrieval system 200 based on RAG and a large language model, such as Figure 4 As shown, including:

[0100] The knowledge base unit 201 is used to parse different types of documents by adopting a long-short mixed granularity method and a preset stripping strategy to obtain text segments of long and short granularities for the different types of documents, and to construct a long-short mixed granularity knowledge base based on the long and short text segments;

[0101] A query unit 202, configured to adjust the user question into a specific query statement by using a multi-round query rewriting mechanism based on a large language model when receiving the user question;

[0102] The retrieval unit 203 is used to use the specific query statement as a retrieval statement, use the retrieval enhancement generation technology RAG to search in the long-short mixed granularity knowledge base, obtain retrieval results, and sort the retrieval results according to the release date of the knowledge in the retrieval results, giving priority to displaying the retrieval results with higher rankings.

[0103] The knowledge base unit 201 adopts a long-short mixed granularity method and a preset stripping strategy to parse different types of documents, including:

[0104] Converting all types of documents into target document data having a hierarchical information representation form;

[0105] The target document data is preliminarily segmented to divide the target document data into a plurality of short-granularity text segments, and incomplete segments in the plurality of short-granularity segments are merged into a plurality of long-granularity text segments.

[0106] The different types of documents include at least one of the following:

[0107] PDF documents, Word documents and Excel documents.

[0108] Optional, hierarchical information representation of the target document data, including:

[0109] The first level is the title, the second level is the sub-title and the third level is the paragraph.

[0110] The knowledge base unit 201 performs preliminary segmentation on the target document data, divides the target document data into a plurality of short-grained text segments, and merges incomplete segments in the plurality of short segments into a plurality of long-grained text segments, including:

[0111] Presetting a maximum length threshold of a chunk, determining the chapter structure of the target document data, and dividing the target document data into a plurality of short-grained text segments according to the maximum length threshold of the chunk based on the chapter structure;

[0112] If the level of the next text segment among the multiple short-grained text segments is lower than the level of the current text segment, and the next text segment does not exceed the maximum length threshold of a chunk after being merged with the current text segment, then the next text segment and the current text segment are merged into a short-grained text segment;

[0113] The completeness of each of the multiple short-grained text fragments is determined, incomplete text fragments from the multiple short-grained text fragments are screened out, and the incomplete fragments are merged into multiple long-grained text fragments.

[0114] Among them, each piece of data in the long-short mixed granularity knowledge base includes:

[0115] At least one short-grained text segment and at least one long-grained text segment.

[0116] The query unit 202 is further used for:

[0117] Acquire the query dialogue information of the user, train the original large language model based on the query dialogue information of the user, adjust the model parameters of the original large language model, and generate a large language model;

[0118] The user's query dialogue information includes: the user's original query information, context information of the original query information, and the user's expected information.

[0119] The application of the present invention improves the accuracy of question-answer retrieval queries.

[0120] Embodiment 3:

[0121] Based on the same inventive concept, the present invention also provides a computer device, which includes a processor and a memory, wherein the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, and is specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding functions, so as to implement the steps of the method in the above embodiment.

[0122] Embodiment 4:

[0123] Based on the same inventive concept, the present invention also provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both a built-in storage medium in a computer device and an extended storage medium supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by a processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the steps of the method in the above embodiment.

[0124] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.

[0125] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0129] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An information retrieval method based on RAG and a large language model, characterized in that: include: The long-short mixed granularity method and the preset stripping strategy are adopted to parse different types of documents to obtain text segments of long and short granularities for the different types of documents, and a long-short mixed granularity knowledge base is constructed based on the text segments of long and short granularities; When receiving a user question, a multi-round query rewriting mechanism is used based on a large language model to adjust the user question into a specific query statement; The specific query statement is used as a search statement, and the search enhancement generation technology RAG is used to search in the long-short mixed granularity knowledge base to obtain search results, and the search results are sorted according to the release date of the knowledge in the search results, and the search results with the highest ranking are displayed first.

2. The information retrieval method according to claim 1, characterized in that: The long-short mixed granularity method and the preset stripping strategy are used to parse different types of documents, including: Converting the documents of all types into target document data having a hierarchical information representation form; The target document data is preliminarily segmented to divide the target document data into a plurality of short-granularity text segments, and incomplete segments in the plurality of short-granularity segments are merged into a plurality of long-granularity text segments.

3. The information retrieval method according to claim 2, characterized in that: The different types of documents include at least one of the following: PDF documents, Word documents and Excel documents.

4. The information retrieval method according to claim 2, characterized in that: The hierarchical information representation form of the target document data includes: The first level is the title, the second level is the sub-title and the third level is the paragraph.

5. The information retrieval method according to claim 2, characterized in that: The preliminary segmentation of the target document data into a plurality of short-grained text segments and merging incomplete segments in the plurality of short segments into a plurality of long-grained text segments includes: Presetting a maximum length threshold of a chunk, determining the chapter structure of the target document data, and dividing the target document data into a plurality of short-grained text segments according to the maximum length threshold of the chunk based on the chapter structure; If the level of the next text segment among the multiple short-grained text segments is lower than the level of the current text segment, and the next text segment does not exceed the maximum length threshold of a chunk after being merged with the current text segment, then the next text segment and the current text segment are merged into a short-grained text segment; The completeness of each of the multiple short-grained text fragments is determined, incomplete text fragments from the multiple short-grained text fragments are screened out, and the incomplete fragments are merged into multiple long-grained text fragments.

6. The information retrieval method according to claim 1, characterized in that: Each piece of data in the long-short mixed granularity knowledge base includes: At least one short-grained text segment and at least one long-grained text segment.

7. The information retrieval method according to claim 1, characterized in that: The method further comprises: Acquire the query dialogue information of the user, train the original large language model based on the query dialogue information of the user, adjust the model parameters of the original large language model, and generate a large language model; The user's query dialogue information includes: the user's original query information, context information of the original query information, and the user's expected information.

8. An information retrieval system based on RAG and large language model, characterized in that: include: The knowledge base unit is used to parse different types of documents by adopting a long-short mixed granularity method and a preset stripping strategy to obtain text segments of long and short granularities for the different types of documents, and to construct a long-short mixed granularity knowledge base based on the long and short text segments; A query unit, configured to, when receiving a user question, adjust the user question into a specific query statement using a multi-round query rewriting mechanism based on a large language model; The retrieval unit is used to use the specific query statement as the retrieval statement, use the retrieval enhancement generation technology RAG to search in the long-short mixed granularity knowledge base, obtain the retrieval results, and sort the retrieval results according to the release date of the knowledge in the retrieval results, giving priority to displaying the retrieval results with the highest ranking.

9. A computer device, characterized in that: include: one or more processors; a processor for executing one or more programs; When the one or more programs are executed by the one or more processors, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed, the method according to any one of claims 1 to 7 is implemented.

Images