A method for generating large language models based on retrieval enhancement

Abstract

A method for generating a large language model based on retrieval enhancement, belonging to the field of computers, includes: retrieving a similar document abstract according to the input question, converting the input question and the document abstract into points in a high-dimensional vector space, retrieving relevant documents and converting them into a structured knowledge graph, generating sub-questions and extracting key information from them, converting it into keywords, and then calling the large language model for few-shot learning to generate an associative prompt; combining the associative prompt and the sub-question to generate a query command; obtaining sub-knowledge graph information by executing the query command and converting it into text and text compression conversion; inputting the converted text and the question into the GPT-4 large language model to generate a natural language answer. The present invention can more accurately process multi-level information and complex data in complex questions, improve the accuracy, efficiency and quality in large-scale data set and complex document processing, provide accurate and coherent answers, and reduce the operating cost and resource consumption.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and particularly relates to a method for generating large language models based on retrieval enhancement. Background Art

[0002] Large Language Models (LLMs) such as ChatGPT and GPT-4 have made significant progress in the field of natural language processing, capable of generating high-quality text, answering questions, and performing various language tasks. However, these models still face a key technical problem, namely the "Hallucination Problem". The hallucination problem refers to the fact that LLMs may generate information that is inconsistent with facts or completely fictional when generating text. This phenomenon can lead to misleading generated content, affecting user trust and the practical application value of the system. The main reasons include noise or incorrect information in the training data, insufficient understanding of complex contexts, limitations in reasoning ability, and the difficulty of fully controlling and interpreting the generated text due to model complexity. The hallucination problem not only affects the reliability and practicality of LLMs but also poses challenges to their applications in key fields such as healthcare, law, and education. Therefore, solving the hallucination problem has become a key task for improving the performance of large language models and expanding their application scope.

[0003] Retrieval Augument Generation (RAG) methods have achieved certain results in solving the hallucination problem of large language models. However, these methods mainly rely on vector similarity for information retrieval and have obvious limitations. On the one hand, in large-scale datasets or complex documents, existing RAG methods often have difficulty effectively connecting different information points. The relationships between information points are complex and diverse, and vector similarity is difficult to fully capture and integrate these relationships, resulting in a lack of coherence and accuracy in the generated content by the model. On the other hand, RAG methods perform poorly when dealing with large documents containing multiple central nodes. Central nodes are the core content of the document, but existing methods are difficult to accurately identify and understand these nodes, resulting in the generated content being difficult to reflect the main idea and key information of the document. These deficiencies significantly limit the application effect of RAG methods in processing large-scale datasets and complex documents. To improve the performance of RAG methods in these scenarios, more advanced technologies need to be introduced in the information retrieval and processing process to enhance the model's ability to connect information points and understand central nodes, thereby improving the processing effect of large-scale datasets and complex documents.

[0004] Current knowledge graph retrieval augmented generation methods need to rely on external APIs (such as gpt-4-turbo) to achieve better results, which brings significant high-cost problems. First of all, using external APIs usually requires paying high fees, especially when called on a large scale or frequently, which will significantly increase the operating costs of the system. For enterprises and research institutions, relying on external APIs for text generation and processing in the long term may lead to budget overruns, restricting the wide application and promotion of this method. Secondly, relying on external APIs will also cause resource waste. The call of external APIs not only consumes a large amount of computing resources, but may also be restricted by external service providers, resulting in an extended system response time and affecting the user experience. To solve these problems, it is necessary to develop the GraphRAG method that does not rely on external APIs. By optimizing the algorithm and model structure, the system can also achieve efficient text generation and processing in the local environment, thereby reducing costs, saving resources, and improving the sustainability and practical application value of the system.

[0005] In the prior art, some representative methods and their disadvantages are introduced as follows:

[0006] 1. ERINE (Enhanced Representation through Knowledge Integration of Entities): ERINE enhances the model's understanding of entities and their relationships by introducing knowledge graph information into the language model. During the pre-training and fine-tuning processes, the model not only learns text data but also obtains rich background knowledge from the knowledge graph. In the question-answering task, ERINE can use the entity information in the knowledge graph to provide more accurate answers. For example, when the user asks "What are the main characteristics of roses?", the model can retrieve the specific characteristics of roses (such as flower color, number of petals, cultivation environment, etc.) from the knowledge graph, and thus generate an accurate and detailed answer.

[0007] Although ERINE can use the knowledge graph to enhance the knowledge background of the generation model, it still has deficiencies in information point connection and central node understanding. The relationship information in the knowledge graph may not be detailed enough or there may be ambiguities, resulting in deviations in the understanding of complex relationships in the generated content.

[0008] 2. COMET (Commonsense Transformers): COMET is a knowledge graph-based language generation model designed to generate commonsense knowledge. By combining the language model and the commonsense knowledge graph, COMET can generate text containing commonsense information. When generating sentences with commonsense, COMET can use the commonsense information in the knowledge graph. For example, when generating "People will use umbrellas on rainy days", the model can retrieve the association between "rain" and "umbrella" from the commonsense knowledge graph and generate a sentence that conforms to common sense.

[0009] COMET relies on common sense knowledge graphs to generate content. However, the coverage of common sense knowledge graphs is limited. For some domain-specific or complex tasks, common sense knowledge graphs may not provide sufficient information support, resulting in the generated content lacking accuracy and relevance. COMET requires a large amount of common sense reasoning during the generation process, which increases the computational complexity. Especially when dealing with complex problems, the multi-level reasoning process will significantly increase the computational burden, affecting the efficiency and performance of the system.

[0010] 3. K-BERT (Knowledge-infused BERT): Based on the BERT model, K-BERT injects knowledge graph information. By adding entity information and relationships of the knowledge graph to the input text, the model can refer to external knowledge during the understanding and generation processes. In reading comprehension tasks, K-BERT can utilize the background information in the knowledge graph to improve the accuracy of answers. For example, when the text mentions "Taj Mahal", the model can learn the detailed background of the Taj Mahal (such as the construction time, location, historical significance, etc.) through the knowledge graph, thus generating more accurate answers.

[0011] In the K-BERT method, the introduction of knowledge graph information may lead to unstable reasoning processes in the model. Especially in multi-hop reasoning tasks, the incorrect information in the knowledge graph will be gradually amplified, affecting the accuracy and coherence of the finally generated content. Due to the need to frequently access and process knowledge graph data during the generation process, the response speed of the K-BERT method is slow. Especially when dealing with complex documents and multi-center node tasks, the real-time performance of the system and the user experience are restricted.

[0012] 4. GraphRAG method: The GraphRAG method is proposed by Microsoft to overcome the limitations of traditional RAG methods. GraphRAG enhances the connection of information points and the understanding ability of central nodes by introducing a graph structure. The GraphRAG method has the following characteristics: ① Graph structure representation: Using knowledge graphs or other graph structures to represent the entities and relationships in the text, making the connections between information points clearer; ② Graph-guided generation: During the generation process, using the graph structure to guide the generation model, enabling it to better understand and integrate complex relationships. When dealing with question-and-answer tasks containing multiple entities and complex relationships, GraphRAG understands the relationships between different entities through the graph structure, thus generating accurate answers. When generating the summary of a long document, GraphRAG identifies the central nodes and main relationships of the document through the graph structure, thus generating a more coherent and comprehensive summary.

[0013] Existing GraphRAG methods and other similar technologies typically rely on external APIs (such as gpt-4-turbo) to achieve efficient text generation. This reliance brings significant cost issues. Especially in large-scale applications, frequent calls to external APIs can lead to high costs, thus limiting the wide application and promotion of these technologies. Summary of the Invention

[0014] In view of the problems existing in the prior art, the present invention provides a retrieval-enhanced large language model generation method. The present invention aims to solve three problems. One is to solve the hallucination problem of large language models when generating content, so as to minimize the generation of false or inaccurate information; the second is to solve the deficiencies of existing RAG methods in information point connection and central node understanding, and improve the effect of large prediction models in processing complex documents; the third is to solve the high cost problem caused by the dependence on external APIs in existing knowledge graph retrieval-enhanced generation methods.

[0015] The technical solutions adopted by the present invention to solve the technical problems are as follows:

[0016] A retrieval-enhanced large language model generation method provided by the present invention includes the following steps:

[0017] Step S1: The file retrieval engine component retrieves document summaries similar to the input question, and uses text embedding technology to convert the input question and document summaries into points in a high-dimensional vector space, and quickly locates relevant documents by calculating the similarity between these vectors;

[0018] Step S2: The knowledge graph builder component converts the retrieved relevant documents into a structured knowledge graph; at the same time, the question parsing and keyword extractor component uses dependency syntax parsing and semantic role annotation technologies to perform semantic analysis on the input question, identify subtasks in the question, generate corresponding sub-questions, extract key information from the input question, and then convert it into keywords; call the large language model for few-shot learning for the keywords to generate relevant association prompt words; combine the generated association prompt words and sub-questions to generate multiple query commands applicable to the knowledge graph; by executing multiple query commands, obtain sub-knowledge graph information of the sub-questions, convert the sub-knowledge graph information into text form, and perform compression conversion on redundant or repeated information in the text;

[0019] Step S3: Input the text obtained by compression conversion and the question into the GPT-4 large language model to generate a natural language answer closely related to the content of the input question.

[0020] Further, in step S1, the file retrieval engine component adopts a small text block retrieval strategy during retrieval.

[0021] Further, in step S1, the text embedding technology calls the embedding model text-embedding-ada-002.

[0022] Further, in step S2, the knowledge graph builder component uses the Mistral local LLM for in-depth semantic analysis, identifies key information in relevant documents, extracts entities in relevant documents through the named entity recognition function of the Mistral local LLM, analyzes semantic relationships in relevant documents, extracts attributes and relationships between entities, and constructs nodes and edges of the knowledge graph; uses the nomic-embed-text embedding model to map entities, attributes, and relationships to a high-dimensional vector space; stores the extracted nodes and edges in the Neo4j graph database.

[0023] Further, the dependency parsing is used to identify the core verb of the sentence and its related syntactic components; the goal of the dependency parsing is to generate a dependency tree, where each word w i is related to its head word h i and the dependency relationship r i , and the parsing task is expressed as:

[0024]

[0025] where T is the optimal dependency tree; τ(S) is the set of all possible dependency trees; Score(h i , w i , r i ) is the score of the word w i being the dependent word of the word h i and having a dependency relationship r i between them.

[0026] Further, the semantic role labeling is used to identify the predicates and arguments of the sentence and assign semantic roles to these arguments; assuming y t represents the semantic role label of the t-th word, the SRL model is represented by the following formula:

[0027] h t = BiLSTM(x t , h t-1 )

[0028] y t = softmax(Wh t + b)

[0029] where: h t is the hidden state based on bidirectional LSTM; x t is the input word embedding or feature vector; y t is the semantic role label of the t-th word.

[0030] Further, in step S2, a template-based method is adopted to convert the sub-knowledge graph information into text form, that is, predefined sentence templates are used, and the structured data is filled into these templates to generate natural language text.

[0031] Further, in step S2, the specific implementation steps of the compression conversion are as follows:

[0032] Train an information compression converter. In the compression stage, according to the given input text S input , generate a compressed sequence C extracted , and the purpose of training is to make the compressed sequence C extracted as close as possible to the actual context sequence C truth , and the loss function used is: the matching loss L between the generated sequence and the target sequence match , the information integrity loss L info , and the context coherence loss L context ; the matching loss L between the generated sequence and the target sequence match is used to measure the matching degree between the compressed sequence C extracted generated by the model and the actual sequence C truth , the information integrity loss L info is used to ensure that the compressed text retains the important information in the original text and avoids missing key content, and the context coherence loss L context is used to ensure that the compressed text is semantically and logically coherent;

[0033] The final comprehensive loss function L is the weighted sum of the above three loss terms:

[0034] L = α·L match + β·L info + γ·L context

[0035] where α, β, and γ are all weight coefficients used to adjust the contributions of the three loss terms to the overall loss.

[0036] Further, the matching loss L between the generated sequence and the target sequence match is calculated by cross-entropy loss; the information integrity loss L info is measured by information entropy or information gain; the context coherence loss L context measures the similarity between the generated sequence and the context by using the latent layer features of a bidirectional language model.

[0037] Furthermore, in step S3, the compressed text and the question are first concatenated into an input sequence, and the GPT-4 large language model processes this input sequence through its deep language understanding ability and generates a most likely answer sequence based on the context.

[0038] The beneficial effects of the present invention are as follows:

[0039] Compared with the prior art, such as the ERINE, COMET, K-BERT, and GraphRAG methods, the present invention has the following advantages:

[0040] 1. Multi-hop question answering and question decomposition;

[0041] The ERINE and COMET methods may face challenges in information connection and context understanding in multi-hop question answering. In contrast, the present invention can more accurately process the multi-level information and complex data in complex questions by decomposing complex questions into multiple simple sub-questions, simultaneously using the large language model to associate keywords and generate association prompt words, as well as Neo4j query commands, thereby improving the accuracy and efficiency in large-scale dataset and complex document processing.

[0042] 2. Compression conversion and information extraction;

[0043] The K-BERT method may have redundant and duplicate information in the process of integrating knowledge graphs. In contrast, the present invention converts the structured knowledge graph information into a concise text form through a trained information compression converter, which can effectively remove redundant and duplicate information. The generated concise text retains the key content, improving the quality and efficiency of information processing.

[0044] 3. Answer generation based on GPT-4;

[0045] The ERINE and COMET have certain limitations in natural language generation. For example, the accuracy and coherence of the generated content may not be ideal. In contrast, the present invention uses the GPT-4 large language model to process the compressed text and the question, and GPT-4 can generate high-quality, accurate, and contextually coherent natural language answers, giving full play to the advantages of GPT-4 in natural language understanding and generation, and providing accurate and coherent answers.

[0046] 4. Optimize the cost and efficiency of the RAG method;

[0047] The GraphRAG method relies on external APIs, resulting in high costs and resource waste. In contrast, the present invention improves the GraphRAG technology. By optimizing the internal algorithm, it solves the problems of high costs and resource waste caused by relying on external APIs in the prior art, reduces the dependence on external APIs, significantly reduces the system operation costs and resource consumption, and at the same time improves the efficiency of processing large-scale data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 It is a flowchart of a method for generating a large language model based on retrieval enhancement provided by the present invention.

[0049] Figure 2 It is a schematic diagram of the keyword extraction process. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] The following further describes the present invention in detail with reference to the accompanying drawings.

[0051] A method for generating a large language model based on retrieval enhancement provided by the present invention aims to reduce the computing cost, enable the system to operate efficiently in a local environment, and reduce the dependence on external resources; by designing a more efficient knowledge retrieval mechanism, the retrieval process of the knowledge graph is made more simple and accurate, ensuring that the system can obtain the latest knowledge information in real time, and improving the accuracy and relevance of the generated content; improving the methods for connecting information points and understanding central nodes, enabling the model to more accurately identify and connect complex relationships, and improving the coherence and accuracy of the generated content, especially when processing complex documents containing multiple central nodes.

[0052] As Figure 1 shown, the specific implementation process of a method for generating a large language model based on retrieval enhancement of the present invention is as follows:

[0053] Step S1: Retrieval;

[0054] First, retrieve files related to the input question according to the input question. Adopting a small text block retrieval strategy, the file retrieval engine component is responsible for receiving the question input by the user and retrieving a document abstract similar to the input question content.

[0055] Then, this file retrieval engine component uses text embedding technology (embedding model text-embedding-ada-002) to convert the question input by the user and the document abstract into points in a high-dimensional vector space, and quickly locates relevant documents (such as Document 1 or Document 2) by calculating the similarity between these vectors.

[0056] Among them, the embedding model "text-embedding-ada-002" used belongs to a part of the OpenAI large language model series. This embedding model "text-embedding-ada-002" converts text (such as words, phrases, or entire texts) into vectors in numerical form, enabling computers to process and understand natural language. The embedding model "text-embedding-ada-002" is widely used in text search, text similarity, and code search, and outperforms the previous most powerful embedding model, Da Vinci, in most tasks, while the price is 99.8% lower than that of Da Vinci.

[0057] Step S2: Graph construction;

[0058] Step S2.1: After retrieving relevant documents, the knowledge graph builder component is responsible for converting the content of the retrieved relevant documents into a structured knowledge graph.

[0059] This knowledge graph builder component uses the Mistral local LLM for in-depth semantic analysis to identify key information in relevant documents, such as entities, attributes, and events, etc. Through the named entity recognition (NER) function of the Mistral local LLM, entities in relevant documents, such as person names, locations, organizations, etc., are extracted, the semantic relationships in relevant documents are analyzed, the attributes and relationships between entities are extracted, and the nodes and edges of the knowledge graph are constructed. In addition, the nomic-embed-text embedding model is used to map entities, attributes, and relationships to a high-dimensional vector space for subsequent similarity comparison and knowledge graph expansion. Finally, the extracted nodes and edges are stored in the Neo4j graph database for querying and updating.

[0060] Step S2.2: Extract keywords;

[0061] While creating the graph, the question parsing and keyword extractor component, as a parallel component, is responsible for deeply understanding the user's input question, identifying subtasks in the question, generating corresponding sub-questions, extracting key information from them, and then converting (rewriting) the extracted key information into keywords. This question parsing and keyword extractor component uses dependency parsing and semantic role labeling (SRL) techniques to ensure that the extracted keywords can accurately reflect the core intention of the user's input question.

[0062] During the process of keyword extraction, the problem analysis and keyword extractor component first need to perform semantic analysis on complex problems to identify the main entities and key relationships. Through semantic analysis, the core verb, related entities, and their relationships in the problem can be clarified, thereby determining which parts of the problem can be processed independently. This semantic analysis method usually relies on Dependency Parsing and Semantic Role Labeling (SRL). Among them, Dependency Parsing helps the system understand the grammatical relationships between words in a sentence, while Semantic Role Labeling clarifies the semantic roles of each entity in the sentence by identifying the predicate and its related arguments.

[0063] Once the subtasks in the problem are identified, the system will generate corresponding sub-problems. These sub-problems are represented in natural language form and can be solved independently. For example, given a complex problem containing multiple entity relationships, take the problem "In the treatment of heart disease, what are the main indications of coronary intervention therapy? In addition, what are the risks and complications of coronary intervention therapy?" as an example. Dependency Parsing first analyzes the grammatical structure of the sentence to identify each component in the sentence and its grammatical relationships with each other.

[0064] For example, as Figure 2 shown, for the problem "In the treatment of heart disease, what are the main indications of coronary intervention therapy? In addition, what are the risks and complications of coronary intervention therapy?", Dependency Parsing will identify the following key components and relationships:

[0065] Predicates: "what are", "what are";

[0066] Subjects: "the main indications of coronary intervention therapy", "the risks and complications of coronary intervention therapy";

[0067] Objects: "the indications of coronary intervention therapy", "risks and complications";

[0068] Modifiers: "In the treatment of heart disease", "of coronary intervention therapy".

[0069] Through Dependency Parsing, the core verb (predicate) of the sentence and its related syntactic components (such as the subject and object) can be identified. In this example, Dependency Parsing helps decompose the original problem into two relatively simple sub-problems:

[0070] "In the treatment of heart disease, what are the main indications of coronary intervention therapy?";

[0071] "In the treatment of heart disease, what are the risks and complications of coronary intervention therapy?".

[0072] Semantic Role Labeling (SRL), when dealing with the two sub-questions "In the treatment of heart disease, what are the main indications for coronary intervention?" and "In the treatment of heart disease, what are the risks and complications of coronary intervention?", respectively labels the predicates and their related arguments in each sentence, and assigns appropriate semantic roles to these arguments.

[0073] Next, the formula is further used to explain in detail how to process the complex question "In the treatment of heart disease, what are the main indications for coronary intervention? Additionally, what are the risks and complications of coronary intervention?" through Dependency Parsing and Semantic Role Labeling (SRL).

[0074] Suppose the sentence S is "In the treatment of heart disease, what are the main indications for coronary intervention? Additionally, what are the risks and complications of coronary intervention?". The goal of dependency parsing is to generate a dependency tree, where each word w i is related to its head word h i and the dependency relation r i . The parsing task can be expressed as:

[0075]

[0076] where T is the optimal dependency tree; τ(S) is the set of all possible dependency trees; Score(h i , w i , r i ) is the score for the word w i being the dependent word of the word h i and having the dependency relation r i .

[0077] After dependency parsing, the system decomposes the question into two sub-questions. In the processing of each sub-question, semantic role labeling is used to identify the predicates and arguments of the sentence and assign semantic roles to these arguments. For example, for the sub-question "In the treatment of heart disease, what are the main indications for coronary intervention?", the goal of semantic role labeling is to label the corresponding semantic roles for the words in the sentence. Suppose y t represents the semantic role label of the t-th word, and the SRL model is represented by the following formula:

[0078] h t = BiLSTM(x t , h t-1 )

[0079] y t= softmax(Wh t + b)

[0080] where: h t is the hidden state based on bidirectional LSTM; x t is the input word embedding or feature vector; y t is the semantic role label of the t-th word.

[0081] For the sub-question "In the treatment of heart disease, what are the main indications for coronary intervention therapy?", the SRL annotation is as follows:

[0082] ● "The main indications for coronary intervention therapy" (Theme);

[0083] ● "What are" (Predicate);

[0084] ● "In the treatment of heart disease" (Location);

[0085] Similarly, for the sub-question "In the treatment of heart disease, what are the risks and complications of coronary intervention therapy?", SRL will identify:

[0086] ● "The risks and complications of coronary intervention therapy" (Theme);

[0087] ● "What are" (Predicate);

[0088] ● "In the treatment of heart disease" (Location);

[0089] These semantic roles help the system correctly understand the keywords of each sub-question, so as to find accurate answers in the knowledge base.

[0090] Finally, for the first sub-question "In the treatment of heart disease, what are the main indications for coronary intervention therapy?", the keyword extraction is:

[0091] K1 = {heart disease, coronary intervention therapy, main indications}

[0092] For the second sub-question "In the treatment of heart disease, what are the risks and complications of coronary intervention therapy?", the keyword extraction is:

[0093] K2 = {heart disease, coronary intervention therapy, risks, complications}

[0094] Step S2.3: Obtain association prompt words;

[0095] For the extracted keywords, the present invention uses an external interface to call a large language model (GPT-4) for few-shot learning. Through its powerful semantic understanding and association capabilities, relevant association prompt words are generated for these keywords. The generated association prompt words not only expand the semantic scope of the keywords but also help capture more relevant information when constructing Neo4j queries.

[0096] For K1 (keywords of the first sub-question), the generated association prompt words L1 are:

[0097] L1 = {coronary artery stenosis, coronary artery occlusion, angina pectoris, acute coronary syndrome, myocardial infarction}

[0098] For K2 (keywords of the second sub-question): The generated association prompt words L2 are:

[0099] L2 = {vascular injury, stent thrombosis, bleeding, infection, myocardial infarction}

[0100] Step S2.4: Obtain the query commands for the knowledge graph;

[0101] Next, multiple query commands applicable to the Neo4j knowledge graph are generated by combining the generated association prompt words and the sub-questions. Each query command expands the query scope through the association prompt words, thereby improving the comprehensiveness and accuracy of information retrieval.

[0102] For the first sub-question, the generated Neo4j query commands respectively query the indication information related to coronary intervention from different perspectives (such as coronary intervention treatment, indications, and associations with heart disease treatment).

[0103] For the second sub-question, the generated Neo4j query commands combine different association prompt words to Ensure retrieval from multiple channels of "Risks and Complications" related to "Coronary Intervention Therapy" that meet the criteria information.

[0104] Execute all the generated Neo4j query commands to obtain the subgraphs related to the sub-questions from the knowledge graph:

[0105] G 1a G 1b G 1c = execute_Neo4j_queries(C 1a ,C 1b ,C 1c )

[0106] G 2a G 2b G 2c = execute_Neo4j_queries(c 2a ,C2b , C 2c )

[0107] By executing multiple query commands, the system retrieves relevant information from different perspectives to obtain the most comprehensive answers. For example, G 1a G 1b G 1c The results in are comprehensively used to identify all the main indications for coronary intervention therapy, while G 2a G 2b G 2c filters out the risks and complications of coronary intervention therapy.

[0108] Step S2.5: Mapping from the knowledge graph to text and text compression;

[0109] After obtaining the sub-knowledge graph information G1 and G2 of the two sub-questions "What are the main indications for coronary intervention therapy in the treatment of heart diseases?" and "What are the risks and complications of coronary intervention therapy in the treatment of heart diseases?", it is necessary to convert this graph information into text form and perform compression conversion on redundant or repetitive information in the text. This process involves techniques of mapping from the knowledge graph to text and text compression.

[0110] For the process of converting the knowledge graph to text, a template-based method is adopted, that is, using predefined sentence templates and filling structured data into these templates to generate natural language text. The advantage of this method is simplicity and controllability. Specific templates such as "[Treatment method] is applicable to [Indications]" are used to fill in the node information in the knowledge graph. For example, for the question "What are the main indications for coronary intervention therapy in the treatment of heart diseases?", the sentence generated by the template method may be: "Coronary intervention therapy is applicable to relieve angina pectoris, acute coronary syndrome, etc.".

[0111] After obtaining the text information, the next step is to perform compression conversion on redundant or repetitive information in the text. In terms of information compression, even though the retriever can retrieve relevant documents from a huge graph database, it still faces the challenge of processing a large amount of retrieved information. Some studies have tried to solve this problem by expanding the context length of the LLM, but the current LLM is still restricted by the context. Therefore, it is particularly important to use information compression conversion to improve the performance of the LLM in this process by reducing information noise, solving the context length limitation, and enhancing the generation effect.

[0112] First, train an information compression converter. In the compression stage, according to the given input text S input , generate a compressed sequence C extracted , and this compressed sequence represents the refined information in the sub-knowledge graph. The purpose of training is to make the compressed sequence C extractedAs close as possible to the actual context sequence C truth , the loss function used is divided into three parts:

[0113] 1. The matching loss L between the generated sequence and the target sequence match ;

[0114] 2. The information integrity loss L info ;

[0115] 3. The context coherence loss L context .

[0116] Specifically, for the first part, the matching loss L between the generated sequence and the target sequence match is mainly used to measure the matching degree between the compressed sequence C generated by the model extracted and the actual sequence C truth , and it can usually be calculated by the cross-entropy loss:

[0117]

[0118] where T is the sequence length, K is the vocabulary size, y t,k and represent the probability distributions of the k-th word in the target sequence and the generated sequence at time step t, respectively.

[0119] For the second part, the information integrity loss L info is designed to ensure that the compressed text retains the important information in the original text and avoid missing key content, and it can usually be measured by entropy or information gain:

[0120]

[0121] where H(C extracted |S input ) is the information entropy of the output compressed sequence C input given the input text S extracted . Through the information integrity loss L info , it can ensure that the generated sequence retains the key information of the original input.

[0122] For the third part, the context coherence loss L context is mainly used to ensure that the compressed text is semantically and logically coherent. It can usually be measured by using the latent layer features of a bidirectional language model (such as BERT) to measure the similarity between the generated sequence and the context:

[0123]

[0124] Among them, BERT(C extracted ) and BERT(S input ) are the BERT encoded representations of the generated sequence and the input text at time step t, respectively. Through the context coherence loss L context , the generated sequence can be encouraged to be contextually consistent with the original input.

[0125] The final comprehensive loss function L is the weighted sum of the above three loss terms:

[0126] L = α·L match + β·L info + γ·L context

[0127] where α, β, and γ are all weight coefficients used to adjust the contributions of the three loss terms to the overall loss. By adjusting these weights, the model can pay more attention to the matching degree between the text and the target sequence, the integrity of information, and the context coherence during the compression process.

[0128] Step S3: Generate a natural language answer;

[0129] Finally, take the text obtained from the compression conversion and the question as inputs, and then input them into the answer generation component, which calls the GPT-4 large language model. The GPT-4 large language model generates a natural language answer closely related to the content of the input question by understanding the input compressed text and the question.

[0130] Specifically, first concatenate the compressed text and the question into an input sequence; the GPT-4 large language model processes this input sequence through its deep language understanding ability and generates a most likely answer sequence according to the context. During the answer generation process, the model will consider the key information in the text and the specific requirements of the question to ensure that the generated answer is both accurate and coherent.

[0131] The above is only the preferred embodiment of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A method for generating a large language model based on retrieval enhancement, characterized in that, It includes the following steps: Step S1: The document retrieval engine component retrieves document summaries similar to the input question, and uses text embedding technology to convert the input question and document summaries into points in a high-dimensional vector space. By calculating the similarity between these vectors, relevant documents can be quickly located; Step S2: The knowledge graph builder component converts the retrieved relevant documents into a structured knowledge graph; at the same time, the question parsing and keyword extractor component uses dependency syntax parsing and semantic role annotation technologies to perform semantic analysis on the input question, identify subtasks in the question, generate corresponding sub-questions, extract key information from the input question, and then convert it into keywords; call the large language model for few-shot learning for the keywords to generate relevant associative prompt words; combine the generated associative prompt words and sub-questions to generate multiple query commands applicable to the knowledge graph; By executing multiple query commands, obtain the sub-knowledge graph information of the sub-questions, convert the sub-knowledge graph information into text form, and perform compression conversion on redundant or repeated information in the text; Step S3: Input the text obtained by compression conversion and the question into the GPT-4 large language model to generate a natural language answer closely related to the content of the input question.

2. The method for generating a large language model based on retrieval enhancement according to claim 1, wherein In step S1, the document retrieval engine component adopts a small text block retrieval strategy during retrieval.

3. A method for generating a large language model based on retrieval enhancement according to claim 1, wherein In step S1, the text embedding technology calls the embedding model text-embedding-ada-002.

4. A method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that, In step S2, the knowledge graph builder component uses the Mistral local LLM for in-depth semantic analysis to identify key information in relevant documents, extracts entities in relevant documents through the named entity recognition function of the Mistral local LLM, analyzes the semantic relationships in relevant documents, extracts attributes and relationships between entities, and constructs the nodes and edges of the knowledge graph; uses the nomic-embed-text embedding model to map entities, attributes, and relationships to a high-dimensional vector space; stores the extracted nodes and edges in the Neo4j graph database.

5. A method for generating a large language model based on retrieval enhancement according to claim 1, wherein The dependency parsing is used to identify the core verb of a sentence and its related syntactic components; the goal of the dependency parsing is to generate a dependency tree, where each word w i is associated with its head word h i and a dependency relation r i , and the parsing task is expressed as: where T is the optimal dependency tree; τ(S) is the set of all possible dependency trees; Score(h i , w i , r i ) is the score of word w i serving as the dependent word of word h i with a dependency relationship r i between them.

6. The method for generating a large language model based on retrieval enhancement according to claim 1, wherein, The semantic role labeling is used to identify the predicates and arguments of a sentence and assign semantic roles to these arguments; assume y t represents the semantic role label of the t-th word, and the SRL model is represented by the following formula: h t = BiLSTM(x t , h t-1 ) y t = softmax(Wh t + b) Where: h t is the hidden state based on bidirectional LSTM; x t is the input word embedding or feature vector; y t is the semantic role label of the t-th word.

7. A method for generating a large language model based on retrieval enhancement according to claim 1, wherein In step S2, a template-based method is used to convert the sub-knowledge graph information into text form, that is, predefined sentence templates are used, and structured data is filled into these templates to generate natural language text.

8. A method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that In step S2, the specific implementation steps of the compression conversion are as follows: Train an information compression converter. During the compression phase, based on the given input text S input , generate a compressed sequence C extracted . The purpose of training is to make the compressed sequence C extracted as close as possible to the actual context sequence C truth . The loss function used is: the matching loss L between the generated sequence and the target sequence match , the information integrity loss L info and the context coherence loss L context ; the matching loss L between the generated sequence and the target sequence match is used to measure the matching degree between the compressed sequence C generated by the model extracted and the actual sequence C truth . The information integrity loss L info is used to ensure that the important information in the original text is retained in the compressed text and key content is not omitted. The context coherence loss L context is used to ensure that the compressed text is semantically and logically coherent; The final comprehensive loss function L is the weighted sum of the above three loss terms: L = α·L match + β·L info + γ·L context Among them, α, β, and γ are all weight coefficients used to adjust the contributions of the three loss terms to the overall loss.

9. A method for generating a large language model based on retrieval enhancement according to claim 8, characterized in that, The matching loss L between the generated sequence and the target sequence match is calculated by the cross-entropy loss; the information integrity loss L info is measured by the information entropy or information gain; the context coherence loss L context measures the similarity between the generated sequence and the context by using the latent layer features of a bidirectional language model.

10. A method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that, In step S3, first connect the compressed text and the question into an input sequence, and the GPT-4 large language model processes this input sequence through its deep language understanding ability and generates a most likely answer sequence according to the context.

Images