Automatic RAG knowledge base construction method and system based on large model

Abstract

The invention belongs to the field of database enhancement, and particularly relates to an automatic RAG knowledge base construction method and system.The method comprises the steps that firstly, an RAG knowledge base is constructed, a user can customize an icon and a name of the RAG knowledge base, meanwhile, a data embedding model and a parallel retrieval mode function module are built in, and a text recall test tool is configured; the data embedding model can convert unstructured data into vectors and store the vectors in the classification vector database; a to-be-retrieved document is uploaded to a knowledge base to be preprocessed, a document text demand space is obtained, an optimal effective recall information fragment set is obtained from parallel retrieval modes of vector retrieval, keyword retrieval, mixed retrieval and the like of the knowledge base by utilizing a segmented TopK recall strategy and a text recall test tool, and the optimal effective recall information fragment set is stored in a database; and finally, obtaining an effective question and answer result according to the document text demand space and the optimal effective recall information fragment set, thereby improving the construction efficiency of the knowledge base and the question and answer accuracy.

Description

Technical Field

[0001] The present invention belongs to the field of database enhancement, and in particular relates to a method and system for automatically constructing a RAG knowledge base based on a large model. Background Art

[0002] When processing image text or simple text input, the big model can show good search results and effectively meet the user's information needs. However, when it comes to document input, the existing technology faces many difficulties. As a common and important knowledge carrier, the complexity of the structure and content of documents puts higher requirements on data search. However, when processing documents, the current big model data search technology often cannot accurately understand and extract key information like processing image text or simple text, and sometimes even gives wrong results due to misinterpretation of document content. This leads to low accuracy of data search and answering through documents, affecting user experience and efficiency of knowledge acquisition. Therefore, there is an urgent need for an innovative method to improve the automatic construction process of RAG knowledge base based on big models, so as to improve the accuracy of document data search and answering, give full play to the advantages of big models in the field of knowledge management, and meet the growing demand for accurate information.

[0003] For example, the patent with authorization announcement number CN117708308B discloses a method and system based on RAG natural language intelligent knowledge base management, the method includes: obtaining a user's natural language query request; based on the natural language query request, determining a target feature vector corresponding to the natural language query request through a first feature extraction model, the first feature extraction model is an embedding model; based on the target feature vector, matching one or more associated corpus blocks in a vectorized corpus, the vectorized corpus is constructed based on several corpus blocks; based on one or more associated corpus blocks, generating an answer text through a language generation model, the language generation model is a large language model; and displaying the answer text to the user.

[0004] For example, the patent application with publication number CN119129739A discloses a method and system for building intelligent customer service interaction based on the large language model RAG framework, including: using multiple division methods to logically divide knowledge documents into fragments, inputting the processed fragments into an embedded model for vectorization, obtaining fragment vectors and storing them in a vector database; vectorizing standard question and answer data and storing them in a vector database; inputting the processed result data into the embedded model for vectorization, obtaining the processed result data vector and storing it in the vector database; converting user consultation questions into semantic vectors through the embedded model, reranking multiple index results once through the reranker algorithm to obtain the final index result; and inputting the final index result into the large language model together with the user consultation question as context to generate a question result.

[0005] The above-mentioned prior art has the following problems: the existing large model data search can give good search results for image text or pure text when processing data input, but when it comes to document input, it cannot be like image text or pure text input, and sometimes even gives wrong results, which makes the accuracy of data search and answering through documents lower. For this reason, the present invention proposes a method and system for automatically constructing a RAG knowledge base based on a large model. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a large-model-based RAG knowledge base automatic construction method and system. First, a RAG knowledge base is constructed, and users can customize its icon and name. At the same time, a data embedding model and a parallel retrieval mode function module are built in, and a text recall test tool is configured. The data embedding model can convert unstructured data into vectors and store them in a classified vector sub-database. The document to be retrieved is uploaded to the knowledge base for preprocessing to obtain the document text demand space. The segmented TopK recall strategy and the text recall test tool are used to obtain the optimal effective recall information fragment set from the knowledge base's vector retrieval, keyword retrieval, hybrid retrieval and other parallel retrieval methods. Finally, an effective question and answer result is obtained based on the document text demand space and the optimal effective recall information fragment set, thereby improving the knowledge base construction efficiency and question and answer accuracy.

[0007] To achieve the above object, the present invention provides the following technical solutions: The automatic construction method of RAG knowledge base based on large model includes: Build the RAG knowledge base and customize the knowledge base icon and name. At the same time, embed the data embedding model and parallel retrieval function module in the RAG knowledge base, and configure the text recall test tool. The data embedding model is used to convert the stored unstructured data into vector representations, and store the converted vector representations in the classified vector sub-database in the knowledge base; the parallel retrieval methods include vector retrieval, keyword retrieval and hybrid retrieval methods; Upload the document to be retrieved to the RAG knowledge base for preprocessing to obtain the document text demand space. According to the document text demand space, the segmented TopK recall strategy and text recall test tool are used to obtain the optimal effective recall information fragment set from the RAG knowledge base. The TopK recall strategy is used to obtain the recall information fragments in the RAG knowledge base whose matching degree with the corresponding requirements in the document text requirement space is greater than the configured recall threshold, and to set the length of the recall information fragments; According to the document text demand space and the optimal set of effective recalled information fragments, effective question answering results are obtained.

[0008] Specifically, the steps of obtaining the optimal effective recall information fragment set include: According to the document text demand space, the set of recalled information fragments and the similarity of the corresponding recalled information fragments are obtained through the set search parameters; Set the Score threshold, and obtain the valid recall information fragment set whose similarity is greater than the Score threshold under the current search parameters according to the obtained recall information fragment set; Modify the set retrieval parameters through the text recall test tool, and repeat the above steps to obtain the optimal set of effective recalled information fragments greater than the score threshold under different retrieval parameters.

[0009] Specifically, the set search parameters include input document sentence or block parameters, single or joint search mode parameters in parallel search mode, number of recalled information segments and length of recalled information segments; The input document sentence or block parameters are used to process the input document content into sentences or blocks, obtain sentence text segments or text blocks, and set the maximum number of sentences contained in a single text block and the number of overlapping sentences between text blocks.

[0010] Specifically, the RAG knowledge base is also equipped with a fine-tuned enhanced disassembly search model; the enhanced disassembly search model is obtained by integrating the positioning disassembly sub-model, the search reasoning sub-model and the synchronous enhanced fine-tuning sub-model; the positioning disassembly sub-model is used for the positioning, disassembly and text conversion of non-text content representation of different content forms of input documents; the search reasoning sub-model is used to use the document text demand space and the optimal effective recall information fragment set to perform reasoning matching of effective question and answer results; the synchronous enhanced fine-tuning sub-model is used to perform synchronous fine-tuning pre-training of the positioning disassembly sub-model and the search reasoning sub-model in real time; The steps to obtain the document text requirement space include: The input document is located, disassembled and transformed according to the positioning disassembly sub-model in the fine-tuned enhanced disassembly search model to obtain the transformed document text space; According to the document text space, by setting the input document sentence or block parameters, the text segmentation or block preprocessing is performed to obtain the document text requirement space.

[0011] Specifically, the steps of fine-tuning the enhanced disassembly search model include: Obtain an input document sequence, ensure that the input document sequence includes at least two content expression forms, locate and decompose the content of the input document sequence according to the corresponding forms, obtain a first input sequence after decomposition, obtain a valid question and answer result based on the decomposition of the first input sequence and historical valid question and answer instructions, and construct a first question and answer training set using the annotated input document sequence, historical valid question and answer instructions, and valid question and answer results; The first question-answering training set is input into the pre-trained positioning disassembly sub-model to obtain a fine-tuned positioning disassembly sub-model. During the fine-tuning process, content positioning disassembly and disassembly content relationship extraction are performed according to the content expression form, and non-text expression content is converted into text content features to obtain the conversion content loss probability and the relationship extraction loss probability; According to the text content features and the corresponding content extraction relations and the relation extraction loss probability, a first information search chain is constructed, and at the same time, the first fine-tuning loss function is constructed using the conversion content loss probability and the relation extraction loss probability.

[0012] Specifically, the steps of strengthening the model fine-tuning training of the disassembly search also include: The first information search chain is input into the pre-trained search reasoning sub-model for fine-tuning to obtain a fine-tuned search reasoning sub-model, and during the fine-tuning process, the text content features in the first information search chain and the optimal effective recall information fragment set obtained from the corresponding node in the RAG knowledge base are used to obtain the effective question-answering result of each fine-tuning and the corresponding second fine-tuning loss function; According to the first fine-tuning loss function and the second fine-tuning loss function, a reward function of the synchronous reinforcement fine-tuning sub-model is constructed, and the positioning disassembly sub-model and the search reasoning sub-model are synchronously fine-tuned in real time to obtain real-time and effective question-answering results.

[0013] Specifically, the steps of real-time synchronization fine-tuning include: Set the first fine-tuning loss function threshold, input each real-time effective question and answer result and the corresponding input document into the positioning and decomposition sub-model for positioning, decomposition and conversion, and synchronously input the converted text content features whose first fine-tuning loss function is less than the first fine-tuning loss function threshold into the search reasoning sub-model for reasoning and matching, to obtain the effective question and answer result and the second fine-tuning loss function; A second fine-tuning loss function threshold is set. When the second fine-tuning loss function is less than the second fine-tuning loss function threshold, the corresponding effective question-answering results and input documents are fed back to the enhanced disassembly search model through the synchronous enhanced fine-tuning sub-model for real-time loop fine-tuning.

[0014] Specifically, the content presentation forms include text, diagrams, pictures, notes, annotations, formulas, and mathematical symbols and formulas; The steps of constructing the first information search chain include: According to different content expression forms in the document, the contextual semantic features of different expression forms in the original document in the disassembly process are located, and the connection relationship chain between the text content features corresponding to different expression forms is constructed; According to the text content features corresponding to different expression forms and the corresponding connection relationship chains, a first information search chain is obtained through a graph algorithm.

[0015] The RAG knowledge base automatic construction system based on the large model includes: database construction module, document processing module and enhanced disassembly reasoning module; Database construction module, used to build RAG knowledge base and customize knowledge base icon and knowledge base name, while building data embedding model and parallel retrieval function in RAG knowledge base, and configuring text recall test tool; The document processing module is used to locate, disassemble and transform the input document, and decompose the document text space obtained after the transformation into sentences or blocks to obtain the document text requirement space; The enhanced disassembly reasoning module includes a segmented recall unit, a reasoning matching unit, and a synchronous optimization unit; The segmented recall unit is used to obtain the optimal effective recall information segment set from the RAG knowledge base according to the document text demand space through the segmented TopK recall strategy and text recall test tool; The reasoning matching unit is used to obtain real-time effective question-answering results by searching the reasoning sub-model according to the document text demand space and the optimal effective recall information fragment set; The synchronous optimization unit fine-tunes and optimizes the enhanced disassembly search model through the synchronous enhanced fine-tuning sub-model based on the real-time effective question and answer results and the input documents.

[0016] A computer-readable storage medium stores computer instructions, which, when executed, execute an automatic construction method of a RAG knowledge base based on a large model.

[0017] Compared with the prior art, the present invention has the following beneficial effects: 1. In view of the deficiencies of the prior art, the present invention significantly improves the accuracy of data search and question answering during document input by constructing a RAG knowledge base and introducing an enhanced disassembly search model. Specifically, firstly, the RAG knowledge base has a built-in data embedding model that can convert unstructured data into vector representations and store them in a classified vector sub-database, thereby achieving efficient indexing and retrieval of complex content; secondly, through the segmented TopK recall strategy and text recall test tool, the retrieval method and retrieval parameters are adjusted in real time to obtain the optimal set of effective recalled information fragments. This process not only takes into account the matching degree between the recalled information fragments and the requirements, but also reduces the interference of irrelevant matching information in the knowledge base, thereby improving the retrieval rate and ensuring the relevance and practicality of the recall results.

[0018] 2. In view of the deficiencies in the prior art, the present invention also strengthens the disassembly search model to accurately locate and parse different content forms of documents, and converts them into unified text content features, and performs deep reasoning and matching on the parsed text features to obtain effective question and answer results, thereby ensuring the accuracy of the answers. At the same time, by synchronously strengthening the fine-tuning sub-model, the parameters of the two sub-models are adjusted in real time according to the effective question and answer results and the input document, so that the knowledge base can handle a variety of content expression forms, enhance the model's ability to understand documents, and further improve the accuracy of document-based data search questions and answers. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of a method for automatically constructing a RAG knowledge base based on a large model according to Embodiment 1 of the present invention; Figure 2 This is an interface diagram of specific settings for vector search in the parallel search method in Example 1 of the present invention; Figure 3 This is a system structure diagram for automatically building a RAG knowledge base based on a large model in Example 2 of the present invention. DETAILED DESCRIPTION

[0020] Example 1

[0021] See also Figure 1 The present invention provides an embodiment: a method for automatically constructing a RAG knowledge base based on a large model, the steps comprising: S1. Build a RAG knowledge base and customize the knowledge base icon and knowledge base name. At the same time, embed the data embedding model and parallel retrieval function module in the RAG knowledge base, and configure the text recall test tool; Furthermore, in the knowledge base constructed in this embodiment, the configuration of the current knowledge base can be set and adjusted, and the information of the knowledge base icon, knowledge base name, knowledge base description, Embedding model and retrieval setting can be modified; Furthermore, in this embodiment, the knowledge base is configured with a text recall test tool to help users test and adjust the recall effect under different retrieval methods and parameter settings; the specific operation process is: The user needs to enter a typical user question in the input box of the corresponding knowledge base interface, click "Test", and then view the recall results. In addition, the text recall test tool provides a recent query function, allowing users to view historical query records, including knowledge base queries triggered by associated applications.

[0022] Furthermore, in the process of creating a knowledge base in this embodiment, the user can use the default knowledge base icon or modify the icon by himself; when creating a knowledge base, the name of the knowledge base must be filled in and the description of the knowledge base must be entered.

[0023] The data embedding model is used to convert the stored unstructured data into vector representations, and store the converted vector representations in a classified vector sub-database in the knowledge base; the parallel retrieval method includes vector retrieval, keyword retrieval and hybrid retrieval methods; Furthermore, in this embodiment, specific settings are made for the specific working processes of the vector search, keyword search and hybrid search in the parallel search mode, specifically: During vector retrieval: When users ask questions about the knowledge base, the user's question is converted into a query vector and compared with the text vector in the knowledge base. The similarity is calculated to find the most relevant text segment; See also Figure 2 , the vector retrieval settings include: Rerank model: The knowledge base uses the Rerank reranking model to semantically rerank the initial search results to optimize the ranking results. After configuring the Rerank model API Key on the knowledge base model integration page, you can enable the Rerank model in the search settings.

[0024] TopK: used to filter text fragments with the highest similarity to user questions. The knowledge base automatically adjusts the number of fragments according to the context window size of the selected model; the default value is 10, and the larger the value, the more text fragments are expected to be recalled.

[0025] Score threshold: used to set the similarity screening criteria for text fragments. Only text fragments exceeding the set score will be recalled. The default value is 0.5. The higher the value, the higher the requirement for the similarity between the text and the question, and the smaller the expected number of recalled texts.

[0026] During the keyword search process: When a user asks a question, the knowledge base indexes the key words in the document, allowing the user to search for keywords and returning relevant text fragments containing these keywords. This is similar to plain text retrieval in a search engine. The specific settings are the same as those for vector retrieval, so I will not go into details here. During hybrid retrieval: Perform full-text search and vector search simultaneously, which can not only find related content by similarity, but also filter results based on exact matching keywords.

[0027] Furthermore, the Rerank model activation process in the hybrid search in this embodiment is the same as the activation process in the above-mentioned vector search; After hybrid retrieval, the system will semantically re-rank the retrieved document results again to optimize the ranking results. The working process of TopK and Score thresholds is the same as that of vector retrieval.

[0028] Further, in this embodiment, after the single or joint vector retrieval in the above-mentioned parallel retrieval method, the recall effect is closely related to the semantic accuracy of the data embedding model. High-precision recall requires the database to understand and process the complex semantic relationship in the high-dimensional vector space to ensure the high relevance of the recall content and the query, which makes the storage accuracy of the data embedding model very high. In this embodiment, the classification algorithm and the attention neural network are used to first classify the stored data, and the attention network is used to set a higher weight for the same category of data so that the same data can be clustered together, and the difference attention weights of different categories of data are set so that different categories of data can be better separated, and the classified data is converted into a vector representation and stored in the data type label node corresponding to the knowledge base. Further, the data type here is defined by the user according to the requirements of the field when creating the database, and the defined label is matched one by one with the storage node of the knowledge base.

[0029] Furthermore, in this embodiment, the data embedding model includes two modes: Chinese mode and multilingual mode; when there is only one language in the user document, the general mode can be selected; when the user document involves multiple languages ​​such as Chinese and English, the multilingual mode can be selected, and the Chinese model and the English model can be selected respectively; of course, in this embodiment, the database can also select the multilingual mode when the document has only one language, and the same Chinese and English models can be selected.

[0030] S2, uploading the document to be retrieved to the RAG knowledge base for preprocessing, obtaining the document text demand space, and obtaining the optimal effective recall information fragment set from the RAG knowledge base through the segmented TopK recall strategy and text recall test tool according to the document text demand space; Furthermore, in this embodiment, the database supports uploading local files, and currently supports TXT, PDF, and DOCX file formats, each file does not exceed 100 MB, and after the file is uploaded, it can be previewed and deleted.

[0031] Furthermore, in this embodiment, the step of obtaining the optimal effective recall information segment set includes: According to the document text demand space, a set of recalled information fragments and similarities of corresponding recalled information fragments are obtained by setting search parameters; Further, the search parameters set in this embodiment include input document sentence or block parameters, single or joint search mode parameters in parallel search mode, number of recalled information segments and length of recalled information segments; The input document sentence or block parameters are used to perform sentence or block processing on the input document content, obtain sentence text segments or text blocks, and set the maximum number of sentences contained in a single text block and the number of sentences overlapping between text blocks.

[0032] Furthermore, in the database search of this embodiment, the user can choose to use sentences or text blocks for block setting. Using sentences for block setting means that the article is segmented according to sentences; using text blocks for block setting means that the text content is divided into several independent parts. The user can set the maximum number of sentences in a text block and the number of overlapping sentences between blocks. Furthermore, this knowledge base defaults to setting the maximum number of sentences to 3 and the number of overlapping sentences between blocks to 1.

[0033] Set the Score threshold, and obtain the valid recall information fragment set whose similarity is greater than the Score threshold under the current search parameters according to the obtained recall information fragment set; The set retrieval parameters are modified through the text recall test tool, and the above steps are repeated to obtain the optimal effective recall information fragment set greater than the Score threshold under different retrieval parameters.

[0034] The TopK recall strategy is used to obtain the recall information segments in the RAG knowledge base whose matching degree with the corresponding demand in the document text demand space is greater than the configured recall threshold, and to set the length of the recall information segments; Furthermore, the specific reasons for setting the information segment length in this embodiment are: Since the context window of a large language model is limited, it is impossible to process all the contents in the knowledge base at one time; therefore, in this embodiment, the method of segmenting the long text into multiple paragraphs is adopted, and then based on the user's question, only those paragraphs that are most relevant to the question are recalled. This is the so-called segmented TopK recall strategy; at the same time, when semantically matching the user's question with these text paragraphs, selecting a suitable paragraph length helps to more accurately find the information in the knowledge base that is most relevant to the question, thereby reducing the interference of irrelevant information.

[0035] Furthermore, in this embodiment, the RAG knowledge base is also configured with an enhanced disassembly search model after fine-tuning training; the enhanced disassembly search model is obtained by integrating the positioning disassembly sub-model, the search reasoning sub-model and the synchronous enhanced fine-tuning sub-model; the positioning disassembly sub-model is used for the positioning, disassembly and text conversion of non-text content representation of different content forms of input documents; the search reasoning sub-model is used for reasoning and matching of effective question and answer results using the document text demand space and the optimal effective recall information fragment set; the synchronous enhanced fine-tuning sub-model is used for real-time synchronous fine-tuning pre-training of the positioning disassembly sub-model and the search reasoning sub-model.

[0036] In the above process, by building a RAG (Retrieval-Augmented Generation) knowledge base and integrating a variety of advanced functional modules, the accuracy of data search and question answering during document input and the user experience are significantly improved; first, users can customize the knowledge base icon, name and description, which enhances the flexibility of personalized configuration and meets the needs of different application scenarios. The built-in data embedding model can efficiently convert unstructured data into vector representation and store it in a classified vector sub-database, ensuring efficient indexing and retrieval of complex content; second, the system introduces parallel retrieval methods, including vector retrieval, keyword retrieval and hybrid retrieval, to provide users with a variety of retrieval options; vector retrieval uses the similarity calculation between the query vector and the text vector to find the most relevant text fragments; keyword retrieval indexes the key words in the document and returns the relevant fragments containing these keywords; hybrid retrieval combines the advantages of the two methods, which can not only find relevant content by similarity, but also filter results based on exact matches; in addition, the application of the Rerank re-ranking model optimizes the semantic sorting of the preliminary retrieval results and improves the relevance and accuracy of the recall results. The settings of TopK parameters and Score thresholds allow users to dynamically adjust the quantity and quality of recalled texts according to specific needs, further improving the adaptability of the system. Furthermore, the positioning disassembly sub-model, search reasoning sub-model and synchronous strengthening fine-tuning sub-model in the enhanced disassembly search model work together to solve the limitations of existing technologies in processing document input. The positioning disassembly sub-model can parse different content forms of documents, such as text, charts, pictures, etc., and convert them into unified text content features. The search reasoning sub-model uses these features to perform inference matching of effective question and answer results. The synchronous strengthening fine-tuning sub-model adjusts the parameters of the two sub-models in real time, allowing the system to continuously learn and improve and provide more accurate answers. This intelligent and adaptive mechanism has greatly improved the accuracy of data search questions and answers, especially providing reliable decision support in complex application scenarios. Finally, the introduction of the text recall test tool helps users intuitively test and adjust the recall effect under different retrieval methods and parameter settings. Users can enter typical questions in the input box, click "Test" to view the recall results, and review historical query records through the recent query function; this instant feedback mechanism not only facilitates users to optimize retrieval parameters, but also improves the transparency and ease of use of the system; based on this, the above process not only overcomes the limitations of existing technologies in processing document input, but also greatly improves the accuracy of data search and answering through intelligent and adaptive mechanisms; in addition, the combination of high-precision data embedding models and flexible parallel retrieval methods ensures a high degree of relevance between the recalled content and the query.

[0037] Furthermore, in this embodiment, the positioning disassembly sub-model, the search reasoning sub-model and the synchronous reinforcement fine-tuning sub-model are all obtained by fine-tuning the existing pre-trained public model through a small sample collected locally, and do not need to be retrained on a large scale; for example, the positioning disassembly sub-model is fine-tuned through the image text recognition algorithm and by adding specific positioning and content form splitting annotations; the search reasoning sub-model is obtained through the existing Chinese-based pre-trained Bert model or o1-preview, o1 and o1-mini and other publicly pre-trained models, and the synchronous reinforcement fine-tuning sub-model is obtained through the reinforcement learning algorithm.

[0038] The step of obtaining the document text requirement space comprises: Positioning, disassembling and transforming the input document according to the positioning and disassembling sub-model in the fine-tuned enhanced disassembling search model to obtain a transformed document text space; According to the document text space, by setting the input document sentence segmentation or block parameters, text segmentation or block preprocessing is performed to obtain the document text requirement space.

[0039] S3. Obtaining effective question-answering results based on the document text demand space and the optimal effective recall information fragment set.

[0040] Furthermore, in this embodiment, the step of strengthening the fine-tuning training of the disassembly search model includes: Obtain an input document sequence, ensure that the input document sequence includes at least two content expression forms, locate and decompose the content of the input document sequence according to the corresponding forms, obtain a decomposed first input sequence, obtain a valid question and answer result based on the decomposed first input sequence and historical valid question and answer instructions, and construct a first question and answer training set using the annotated input document sequence, historical valid question and answer instructions, and valid question and answer results; Furthermore, in this embodiment, the content is expressed in the form of text, charts, pictures, notes, annotations, formulas and mathematical symbols.

[0041] Input the first question-answer training set into the pre-trained positioning disassembly sub-model to obtain a fine-tuned positioning disassembly sub-model, perform content positioning disassembly and disassembly content relationship extraction according to the content expression form during the fine-tuning process, and convert non-textual expression content into textual content features to obtain conversion content loss probability and relationship extraction loss probability; According to the text content features and the corresponding content extraction relations and the relation extraction loss probability, a first information search chain is constructed, and a first fine-tuning loss function is constructed using the conversion content loss probability and the relation extraction loss probability; Furthermore, in this embodiment, the steps of constructing the first information search chain include: According to different content expression forms in the document, contextual semantic features of different expression forms in the original document during the disassembly process are located, and a connection relationship chain between the text content features corresponding to the different expression forms is constructed; According to the text content features corresponding to different expression forms and the corresponding connection relationship chains, a first information search chain is obtained through a graph algorithm.

[0042] The first information search chain is input into the pre-trained search reasoning sub-model for fine-tuning to obtain a fine-tuned search reasoning sub-model, and during the fine-tuning process, the text content features in the first information search chain and the optimal effective recall information fragment set obtained from the corresponding node in the RAG knowledge base are used to obtain the effective question-answering result of each fine-tuning and the corresponding second fine-tuning loss function; A reward function for the synchronously reinforced fine-tuning sub-model is constructed based on the first fine-tuning loss function and the second fine-tuning loss function, and the positioning disassembly sub-model and the search reasoning sub-model are synchronously fine-tuned in real time to obtain real-time and effective question-answering results.

[0043] Furthermore, in this embodiment, the step of real-time synchronization fine-tuning includes: Set the first fine-tuning loss function threshold, input each real-time effective question and answer result and the corresponding input document into the positioning and decomposition sub-model for positioning, decomposition and conversion, and synchronously input the converted text content features whose first fine-tuning loss function is less than the first fine-tuning loss function threshold into the search reasoning sub-model for reasoning and matching, to obtain the effective question and answer result and the second fine-tuning loss function; A second fine-tuning loss function threshold is set. When the second fine-tuning loss function is less than the second fine-tuning loss function threshold, the corresponding effective question-answering results and input documents are fed back to the enhanced disassembly search model through the synchronous enhanced fine-tuning sub-model for real-time loop fine-tuning.

[0044] This process significantly improves the data search question-answering accuracy and the system's adaptability during document input by building a RAG knowledge base and integrating a fine-tuned reinforced disassembly search model. First, the reinforced disassembly search model consists of a positioning disassembly sub-model, a search reasoning sub-model, and a synchronous reinforced fine-tuning sub-model. These sub-models are fine-tuned on small samples based on existing pre-trained public models, avoiding the cost and time consumption of large-scale retraining. For example, the positioning and decomposition sub-model uses the image text recognition algorithm and adds specific positioning and content form decomposition annotations for fine-tuning. It can efficiently parse various content forms in the document, such as text, charts, pictures, notes, annotations, formulas, etc., and convert them into unified text content features; secondly, the search reasoning sub-model can perform inference matching of effective question and answer results for the document text demand space and the optimal effective recall information fragment set after fine-tuning through the existing pre-trained Bert model or other public models, ensuring the accuracy and relevance of the answers; the synchronous reinforcement fine-tuning sub-model uses the reinforcement learning algorithm to continuously adjust and optimize the parameters of the two sub-models in real-time interaction, so that the system can continuously improve according to user feedback and provide more accurate answers; in the process of obtaining the document text demand space, the positioning and decomposition sub-model locates, decomposes and transforms the input document to obtain the converted document text space; then, through the set input document sentence or block parameters, text segmentation or block preprocessing is performed to obtain the document text demand space. This processing method not only improves the efficiency of text processing, but also enhances the ability to understand and parse complex document structures; further, by constructing the first information search chain, the contextual semantic features of different content expression forms are connected to form a comprehensive information retrieval chain, so that the system can better understand the relationship between different content forms, thereby improving the comprehensiveness and accuracy of information retrieval; the construction of the first fine-tuning loss function and the second fine-tuning loss function, combined with the reward function of the synchronous reinforcement fine-tuning sub-model, realizes the real-time synchronous fine-tuning of the positioning disassembly sub-model and the search reasoning sub-model, ensuring the stability and efficiency of the system in practical applications; in summary, this process not only greatly improves the accuracy of data search question and answer by introducing the reinforcement disassembly search model, combined with an efficient fine-tuning training method; it also enhances the system's adaptability and user experience; especially for complex documents containing multiple content expression forms, the system can accurately parse and provide high-quality question and answer results.

[0045] Example 2

[0046] See also Figure 3 ,Another embodiment provided by the present invention: A RAG knowledge base automatic construction system based on a large model, comprising: a database construction module, a document processing module and an enhanced disassembly reasoning module; The database construction module is used to construct a RAG knowledge base and customize the knowledge base icon and knowledge base name, while building a data embedding model and parallel retrieval mode function in the RAG knowledge base, and configuring a text recall test tool; The document processing module is used to locate, disassemble and transform the input document, and decompose the document text space obtained after the transformation into sentences or blocks to obtain the document text requirement space; The enhanced disassembly and reasoning module is used for locating, disassembling, text conversion and reasoning matching of different content expressions of input documents; The enhanced disassembly reasoning module includes a segmented recall unit, a reasoning matching unit and a synchronous optimization unit; The segmented recall unit is used to obtain the optimal effective recall information segment set from the RAG knowledge base through the segmented TopK recall strategy and the text recall test tool according to the document text demand space; The reasoning matching unit is used to obtain real-time effective question-answering results by searching the reasoning sub-model according to the document text demand space and the optimal effective recall information fragment set; The synchronous optimization unit fine-tunes and optimizes the enhanced disassembly search model through the synchronous enhanced fine-tuning sub-model according to the real-time effective question and answer results and the input documents.

[0047] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in the field may also change, modify, replace and modify the above-mentioned embodiments without departing from the scope of protection of the purpose of the present invention and the claims, and all of these are within the protection of the present invention.

[0048] If the disclosed technical solution involves personal information, the product using the disclosed technical solution has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the disclosed technical solution involves sensitive personal information, the product using the disclosed technical solution has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A method for automatically constructing a RAG knowledge base based on a large model, characterized in that: include: Build the RAG knowledge base and customize the knowledge base icon and name. At the same time, embed the data embedding model and parallel retrieval function module in the RAG knowledge base, and configure the text recall test tool. The data embedding model is used to convert the stored unstructured data into vector representations, and store the converted vector representations in a classified vector sub-database in the knowledge base; the parallel retrieval method includes vector retrieval, keyword retrieval and hybrid retrieval methods; Upload the document to be retrieved to the RAG knowledge base for preprocessing to obtain the document text demand space, and obtain the optimal effective recall information fragment set from the RAG knowledge base through the segmented TopK recall strategy and text recall test tool according to the document text demand space; The TopK recall strategy is used to obtain the recall information segments in the RAG knowledge base whose matching degree with the corresponding demand in the document text demand space is greater than the configured recall threshold, and to set the length of the recall information segments; According to the document text demand space and the optimal effective recall information fragment set, an effective question-answering result is obtained.

2. The method for automatically constructing a RAG knowledge base based on a large model as claimed in claim 1, characterized in that: The step of obtaining the optimal effective recall information fragment set comprises: According to the document text demand space, a set of recalled information fragments and similarities of corresponding recalled information fragments are obtained by setting search parameters; Set the Score threshold, and obtain the valid recall information fragment set whose similarity is greater than the Score threshold under the current search parameters according to the obtained recall information fragment set; The set retrieval parameters are modified through the text recall test tool, and the above steps are repeated to obtain the optimal effective recall information fragment set greater than the Score threshold under different retrieval parameters.

3. The method for automatically constructing a RAG knowledge base based on a large model as claimed in claim 2, characterized in that: The set search parameters include input document sentence or block parameters, single or joint search mode parameters in parallel search mode, number of recalled information segments and length of recalled information segments; The input document sentence or block parameters are used to perform sentence or block processing on the input document content, obtain sentence text segments or text blocks, and set the maximum number of sentences contained in a single text block and the number of sentences overlapping between text blocks.

4. The method for automatically constructing a RAG knowledge base based on a large model as claimed in claim 3, characterized in that: The RAG knowledge base is also configured with a fine-tuned enhanced disassembly search model; the enhanced disassembly search model is obtained by integrating the positioning disassembly sub-model, the search reasoning sub-model and the synchronous enhanced fine-tuning sub-model; the positioning disassembly sub-model is used for positioning, disassembling and text conversion of non-text content representation of different content forms of input documents; the search reasoning sub-model is used for reasoning and matching of effective question and answer results using the document text demand space and the optimal effective recall information fragment set; the synchronous enhanced fine-tuning sub-model is used for real-time synchronous fine-tuning pre-training of the positioning disassembly sub-model and the search reasoning sub-model; The step of obtaining the document text requirement space comprises: Positioning, disassembling and transforming the input document according to the positioning and disassembling sub-model in the fine-tuned enhanced disassembling search model to obtain a transformed document text space; According to the document text space, by setting the input document sentence segmentation or block parameters, text segmentation or block preprocessing is performed to obtain the document text requirement space.

5. The method for automatically constructing a RAG knowledge base based on a large model as claimed in claim 4, characterized in that: The step of fine-tuning the enhanced disassembly search model includes: Obtain an input document sequence, ensure that the input document sequence includes at least two content expression forms, locate and decompose the content of the input document sequence according to the corresponding forms, obtain a decomposed first input sequence, obtain a valid question and answer result based on the decomposed first input sequence and historical valid question and answer instructions, and construct a first question and answer training set using the annotated input document sequence, historical valid question and answer instructions, and valid question and answer results; Input the first question-answer training set into the pre-trained positioning disassembly sub-model to obtain a fine-tuned positioning disassembly sub-model, perform content positioning disassembly and disassembly content relationship extraction according to the content expression form during the fine-tuning process, and convert non-textual expression content into textual content features to obtain conversion content loss probability and relationship extraction loss probability; According to the text content features and the corresponding content extraction relations and the relation extraction loss probability, a first information search chain is constructed, and a first fine-tuning loss function is constructed using the conversion content loss probability and the relation extraction loss probability.

6. The method for automatically constructing a RAG knowledge base based on a large model as claimed in claim 5, characterized in that: The step of strengthening the fine-tuning training of the disassembly search model also includes: The first information search chain is input into the pre-trained search reasoning sub-model for fine-tuning to obtain a fine-tuned search reasoning sub-model, and during the fine-tuning process, the text content features in the first information search chain and the optimal effective recall information fragment set obtained from the corresponding node in the RAG knowledge base are used to obtain the effective question-answering result of each fine-tuning and the corresponding second fine-tuning loss function; A reward function for the synchronously reinforced fine-tuning sub-model is constructed based on the first fine-tuning loss function and the second fine-tuning loss function, and the positioning disassembly sub-model and the search reasoning sub-model are synchronously fine-tuned in real time to obtain real-time and effective question-answering results.

7. The method for automatically constructing a RAG knowledge base based on a large model as claimed in claim 6, characterized in that: The step of real-time synchronization fine-tuning includes: Set the first fine-tuning loss function threshold, input each real-time effective question and answer result and the corresponding input document into the positioning and decomposition sub-model for positioning, decomposition and conversion, and synchronously input the converted text content features whose first fine-tuning loss function is less than the first fine-tuning loss function threshold into the search reasoning sub-model for reasoning and matching, to obtain the effective question and answer result and the second fine-tuning loss function; A second fine-tuning loss function threshold is set. When the second fine-tuning loss function is less than the second fine-tuning loss function threshold, the corresponding effective question-answering results and input documents are fed back to the enhanced disassembly search model through the synchronous enhanced fine-tuning sub-model for real-time loop fine-tuning.

8. The method for automatically constructing a RAG knowledge base based on a large model as claimed in claim 7, characterized in that: The content expression forms include text, charts, pictures, notes, annotations, formulas and mathematical symbols; The steps of constructing the first information search chain include: According to different content expression forms in the document, contextual semantic features of different expression forms in the original document during the disassembly process are located, and a connection relationship chain between the text content features corresponding to the different expression forms is constructed; According to the text content features corresponding to different expression forms and the corresponding connection relationship chains, a first information search chain is obtained through a graph algorithm.

9. A large model-based RAG knowledge base automatic construction system, which is used to implement the large model-based RAG knowledge base automatic construction method according to any one of claims 1 to 8, characterized in that: include: Database construction module, enhanced disassembly reasoning module and document processing module; The database construction module is used to construct a RAG knowledge base and customize the knowledge base icon and knowledge base name, while building a data embedding model and parallel retrieval mode function in the RAG knowledge base, and configuring a text recall test tool; The document processing module is used to locate, disassemble and transform the input document, and decompose the document text space obtained after the transformation into sentences or blocks to obtain the document text requirement space; The enhanced disassembly reasoning module includes a segmented recall unit, a reasoning matching unit and a synchronous optimization unit; The segmented recall unit is used to obtain the optimal effective recall information segment set from the RAG knowledge base through the segmented TopK recall strategy and the text recall test tool according to the document text demand space; The reasoning matching unit is used to obtain real-time effective question-answering results by searching the reasoning sub-model according to the document text demand space and the optimal effective recall information fragment set; The synchronous optimization unit fine-tunes and optimizes the enhanced disassembly search model through the synchronous enhanced fine-tuning sub-model according to the real-time effective question and answer results and the input documents.

10. A computer-readable storage medium, characterized in that: Computer instructions are stored thereon, and when the computer instructions are executed, the method for automatically constructing a RAG knowledge base based on a large model as described in any one of claims 1 to 8 is executed.

Images