Large language model generation method based on retrieval enhancement

Abstract

A large language model generation method based on retrieval enhancement belongs to the field of computers and comprises the following steps: retrieving similar document abstracts according to input questions, converting the input questions and the document abstracts into points in a high-dimensional vector space, retrieving related documents and converting the related documents into a structured knowledge graph, generating sub-questions and extracting key information from the sub-questions, then, calling a large language model to carry out few-sample learning so as to generate associative cue words; generating a query command in combination with the associative cue word and the sub-question; obtaining sub-knowledge graph information by executing a query command, converting the sub-knowledge graph information into a text, and compressing and converting the text; and inputting the converted text and the question into a GPT-4 large language model to generate a natural language answer. According to the method, multi-level information and complex data in complex questions can be processed more accurately, the accuracy, efficiency and quality in large-scale data set and complex document processing are improved, accurate and coherent answers are provided, and the operation cost and resource consumption are reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a large language model generation method based on retrieval enhancement. Background Art

[0002] Large Language Models (LLMs) such as ChatGPT and GPT-4 have made significant progress in the field of natural language processing, and are able to generate high-quality text, answer questions, and perform various language tasks. However, these models still face a key technical problem, namely the "Hallucination Problem". The hallucination problem refers to the fact that LLM may generate information that is inconsistent with the facts or completely fictitious when generating text. This phenomenon can cause the generated content to be misleading, affecting user trust and the practical application value of the system. The main reasons include noise or false information in the training data, insufficient understanding of complex contexts, limitations in reasoning ability, and difficulty in fully controlling and interpreting generated text due to model complexity. The hallucination problem not only affects the reliability and practicality of LLM, but also poses challenges to its application in key fields such as medicine, law, and education. Therefore, solving the hallucination problem has become a key task to improve the performance of large language models and expand their scope of application.

[0003] Retrival Augment Generation (RAG) methods have achieved certain results in solving the hallucination problem of large language models. However, these methods mainly rely on vector similarity for information retrieval, which has obvious limitations. On the one hand, in large-scale datasets or complex documents, existing RAG methods often find it difficult to effectively connect different information points. The relationships between information points are complex and diverse, and vector similarity is difficult to fully capture and integrate these relationships, resulting in the model's lack of coherence and accuracy when generating content. On the other hand, RAG methods perform poorly when processing large documents containing multiple central nodes. Central nodes are the core content of the document, but existing methods find it difficult to accurately identify and understand these nodes, resulting in the difficulty of generating content that reflects the main purpose and key information of the document. These deficiencies significantly limit the application effect of RAG methods in large-scale datasets and complex document processing. In order to improve the performance of RAG methods in these scenarios, it is necessary to introduce more advanced technologies in the information retrieval and processing process to enhance the model's ability to connect information points and understand central nodes, thereby improving the effect of large-scale datasets and complex document processing.

[0004] Current knowledge graph retrieval enhancement generation methods need to rely on external APIs (such as gpt-4-turbo) to achieve good results, which brings about significant high cost issues. First, the use of external APIs usually requires high fees, especially when called on a large scale or frequently, which significantly increases the operating costs of the system. For enterprises and research institutions, long-term reliance on external APIs for text generation and processing may lead to budget overruns, limiting the widespread application and promotion of this method. Secondly, reliance on external APIs will also lead to resource waste. The call of external APIs not only consumes a lot of computing resources, but may also be restricted by external service providers, resulting in longer system response time and affecting user experience. In order to solve these problems, it is necessary to develop a GraphRAG method that does not rely on external APIs. By optimizing the algorithm and model structure, the system can achieve efficient text generation and processing in a local environment, thereby reducing costs, saving resources, and improving the sustainability and practical application value of the system.

[0005] In the prior art, some representative methods and their shortcomings are introduced as follows:

[0006] 1. ERINE (Enhanced Representation through kNowledge Integration of Entities): ERINE enhances the model's understanding of entities and their relationships by introducing knowledge graph information into the language model. During the pre-training and fine-tuning process, the model not only learns text data, but also acquires rich background knowledge from the knowledge graph. In question-answering tasks, ERINE can use entity information in the knowledge graph to provide more accurate answers. For example, when a user asks, "What are the main characteristics of a rose?", the model can retrieve the specific characteristics of a rose (such as flower color, number of petals, cultivation environment, etc.) from the knowledge graph to generate an accurate and detailed answer.

[0007] Although ERINE can use knowledge graphs to enhance the knowledge background of the generative model, it still has shortcomings in connecting information points and understanding central nodes. The relationship information in the knowledge graph may not be detailed enough or ambiguous, resulting in deviations in the understanding of complex relationships in the generated content.

[0008] 2. COMET (Commonsense Transformers): COMET is a language generation model based on knowledge graphs, which aims to generate common sense knowledge. By combining language models and common sense knowledge graphs, COMET is able to generate text containing common sense information. When generating sentences with common sense, COMET can use the common sense information in the knowledge graph. For example, to generate "People use umbrellas on rainy days", the model can retrieve the association between "rain" and "umbrella" from the common sense knowledge graph and generate a sentence that conforms to common sense.

[0009] COMET relies on common sense knowledge graphs to generate content, but the coverage of common sense knowledge graphs is limited. For some domain-specific or complex tasks, common sense knowledge graphs may not provide sufficient information support, resulting in the lack of accuracy and relevance of the generated content. COMET requires a lot of common sense reasoning during the generation process, which increases the computational complexity. Especially when dealing with complex problems, the multi-level reasoning process will significantly increase the computational burden and affect the efficiency and performance of the system.

[0010] 3. K-BERT (Knowledge-infusedBERT): K-BERT injects knowledge graph information based on the BERT model. By adding entity information and relationships of the knowledge graph to the input text, the model can refer to external knowledge during the understanding and generation process. In reading comprehension tasks, K-BERT can use the background information in the knowledge graph to improve the accuracy of the answer. For example, if the text mentions "Taj Mahal", the model can understand the detailed background of the Taj Mahal (such as construction time, location, historical significance, etc.) through the knowledge graph, thereby generating a more accurate answer.

[0011] In the K-BERT method, the introduction of knowledge graph information may lead to instability in the model's reasoning process. Especially in multi-hop reasoning tasks, the erroneous information in the knowledge graph will be gradually amplified, affecting the accuracy and coherence of the final generated content. Due to the need to frequently access and process knowledge graph data during the generation process, the K-BERT method has a slow response speed. Especially when dealing with complex documents and multi-center node tasks, the system's real-time performance and user experience are limited.

[0012] 4. GraphRAG method: The GraphRAG method was proposed by Microsoft to overcome the limitations of the traditional RAG method. GraphRAG enhances the ability to connect information points and understand central nodes by introducing graph structures. The GraphRAG method has the following characteristics: ① Graph structure representation: Use knowledge graphs or other graph structures to represent entities and relationships in the text, making the connection between information points clearer; ② Graph-guided generation: During the generation process, use the graph structure to guide the generation model so that it can better understand and integrate complex relationships. When dealing with question-answering tasks involving multiple entities and complex relationships, GraphRAG understands the relationship between different entities through the graph structure, thereby generating accurate answers. When generating summaries of long documents, GraphRAG identifies the central nodes and main relationships of the document through the graph structure, thereby generating a more coherent and comprehensive summary.

[0013] Existing GraphRAG methods and other similar techniques usually rely on external APIs (such as gpt-4-turbo) to achieve efficient text generation. This reliance brings significant cost issues, especially when applied on a large scale, where frequent calls to external APIs will result in high costs, thus limiting the widespread application and promotion of these techniques. Summary of the invention

[0014] In view of the problems existing in the prior art, the present invention provides a method for generating a large language model based on retrieval enhancement. The present invention aims to solve three problems: first, to solve the problem of hallucination when the large language model generates content, so as to minimize the generation of false or inaccurate information; second, to solve the shortcomings of the existing RAG method in information point connection and central node understanding, and improve the effect of the large prediction model in processing complex documents; third, to solve the high cost problem caused by the existing knowledge graph retrieval enhancement generation method relying on external API.

[0015] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0016] The present invention provides a method for generating a large language model based on retrieval enhancement, comprising the following steps:

[0017] Step S1: The document retrieval engine component retrieves document summaries similar to the input question based on the question, and uses text embedding technology to convert the input question and document summary into points in a high-dimensional vector space, and quickly locates relevant documents by calculating the similarity between these vectors;

[0018] Step S2: The knowledge graph builder component converts the retrieved relevant documents into a structured knowledge graph; at the same time, the question parsing and keyword extractor component uses dependency syntax parsing and semantic role labeling technology to perform semantic analysis on the input question, identify the subtasks in the question, generate corresponding subquestions, and extract key information from the input question, and then convert it into keywords; call the large language model for few-sample learning for the keywords to generate relevant associative prompt words; combine the generated associative prompt words and subquestions to generate multiple query commands suitable for the knowledge graph; by executing multiple query commands, obtain the sub-knowledge graph information of the subquestions, convert the sub-knowledge graph information into text form, and compress and convert the redundant or repeated information in the text;

[0019] Step S3: Input the compressed converted text and question into the GPT-4 large language model to generate a natural language answer that is closely related to the input question content.

[0020] Furthermore, in step S1, the file retrieval engine component adopts a small text block retrieval strategy during retrieval.

[0021] Furthermore, in step S1, the text embedding technology calls the embedding model text-embedding-ada-002.

[0022] Furthermore, in step S2, the knowledge graph builder component uses the mistral local LLM to perform deep semantic analysis, identify key information in relevant documents, extract entities in relevant documents through the named entity recognition function of the mistral local LLM, analyze the semantic relationships in relevant documents, extract attributes and relationships between entities, and construct nodes and edges of the knowledge graph; use the nomic-embed-text embedding model to map entities, attributes and relationships to a high-dimensional vector space; and store the extracted nodes and edges in the Neo4j graph database.

[0023] Furthermore, the dependency parsing is used to identify the core verbs of a sentence and their related syntactic components; the goal of the dependency parsing is to generate a dependency tree, where each word w i With its central word h i and dependency relationship i , the parsing task is expressed as:

[0024]

[0025] Where T is the optimal dependency tree; τ(S) is the set of all possible dependency trees; Score(h i ,w i ,r i ) is the word w i As a word hi , and there is a dependency relationship between them. i score.

[0026] Furthermore, the semantic role labeling is used to identify the predicate and arguments of a sentence and assign semantic roles to these arguments; assuming y t represents the semantic role label of the t-th word. The SRL model is expressed by the following formula:

[0027] h t =BiLSTM(x t ,h t-1 )

[0028] y t =softmax(Wh t +b)

[0029] Where: h t is the hidden state based on bidirectional LSTM; x t is the input word embedding or feature vector; y t is the semantic role label of the t-th word.

[0030] Furthermore, in step S2, a template-based method is used to convert the sub-knowledge graph information into text form, that is, predefined sentence templates are used and structured data are filled into these templates to generate natural language text.

[0031] Furthermore, in step S2, the specific implementation steps of the compression conversion are as follows:

[0032] Train an information compression transformer. In the compression phase, given the input text S input , generating a compressed sequence C extracted The purpose of training is to make the compressed sequence C extracted As close as possible to the actual context sequence C truth , the loss function used is: the matching loss L between the generated sequence and the target sequence match , information integrity loss L info and contextual coherence loss L context ; The matching loss L between the generated sequence and the target sequence match Used to measure the compressed sequence C generated by the model extracted With the actual sequence C truth The matching degree, the information integrity loss L info It is used to ensure that the compressed text retains important information in the original text and avoids missing key content. The context coherence loss L context Used to ensure that the compressed text is semantically and logically coherent;

[0033] The final comprehensive loss function L is the weighted sum of the above three loss terms:

[0034] L=α·L match +β·L info +γ·L context

[0035] Among them, α, β, and γ are weight coefficients used to adjust the contribution of the three loss terms to the overall loss.

[0036] Furthermore, the matching loss L between the generated sequence and the target sequence is match Calculated by cross entropy loss; the information integrity loss L info It is measured by information entropy or information gain; the context coherence loss L context The similarity between the generated sequence and the context is measured by using the latent layer features of the bidirectional language model.

[0037] Furthermore, in step S3, the compressed text and question are first connected into an input sequence. The GPT-4 large language model processes this input sequence through its deep language understanding ability and generates a most likely answer sequence based on the context.

[0038] The beneficial effects of the present invention are:

[0039] Compared with the prior art, such as ERINE, COMET, K-BERT and GraphRAG methods, the present invention has the following advantages:

[0040] 1. Multi-hop question answering and question decomposition;

[0041] ERINE and COMET methods may face challenges in information connection and context understanding in multi-hop question answering. In contrast, the present invention can more accurately process multi-level information and complex data in complex questions by decomposing complex questions into multiple simple sub-questions and using a large language model to associate keywords and generate association prompts and Neo4j query commands, thereby improving accuracy and efficiency in large-scale data sets and complex document processing.

[0042] 2. Compression conversion and information extraction;

[0043] The K-BERT method may have redundant and repeated information in the process of knowledge graph integration. In contrast, the present invention converts structured knowledge graph information into a concise text form through a trained information compression converter, which can effectively remove redundant and repeated information. The generated concise text retains the key content and improves the quality and efficiency of information processing.

[0044] 3. Answer generation based on GPT-4;

[0045] ERINE and COMET have certain limitations in natural language generation, such as the accuracy and coherence of the generated content may not be ideal. In contrast, the present invention uses the GPT-4 large language model to process compressed text and questions, and uses GPT-4 to generate high-quality, accurate and contextually coherent natural language answers, giving full play to the advantages of GPT-4 in natural language understanding and generation, and providing accurate and coherent answers.

[0046] 4. Optimize the cost and efficiency of the RAG method;

[0047] The GraphRAG method relies on external APIs, resulting in high costs and resource waste. In contrast, the present invention improves the GraphRAG technology, and solves the high cost and resource waste problems caused by reliance on external APIs in the prior art by optimizing the internal algorithm, reduces the reliance on external APIs, significantly reduces system operation costs and resource consumption, and improves the efficiency of processing large-scale data sets. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A flowchart of a method for generating a large language model based on retrieval enhancement provided by the present invention.

[0049] Figure 2 Schematic diagram of the keyword extraction process. DETAILED DESCRIPTION

[0050] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0051] The present invention provides a large language model generation method based on retrieval enhancement, which aims to reduce computing costs, enable the system to run efficiently in a local environment, and reduce dependence on external resources; by designing a more efficient knowledge retrieval mechanism, the retrieval process of the knowledge graph is made simpler and more accurate, ensuring that the system can obtain the latest knowledge information in real time and improve the accuracy and relevance of the generated content; improving the method of connecting information points and understanding central nodes, so that the model can more accurately identify and connect complex relationships, and improve the coherence and accuracy of the generated content, especially when processing complex documents containing multiple central nodes.

[0052] like Figure 1 As shown, the specific implementation process of the method for generating a large language model based on retrieval enhancement of the present invention is as follows:

[0053] Step S1: Retrieval;

[0054] First, the documents related to the input question are retrieved according to the input question. Using the small text block retrieval strategy, the document retrieval engine component is responsible for receiving the user's input question and retrieving the document summaries similar to the input question content.

[0055] Then, the document retrieval engine component uses text embedding technology (embedding model text-embedding-ada-002) to convert the user-input question and document summary into points in a high-dimensional vector space, and quickly locates relevant documents (such as document 1 or document 2) by calculating the similarity between these vectors.

[0056] Among them, the embedding model "text-embedding-ada-002" used is part of OpenAI's large language model series. The embedding model "text-embedding-ada-002" converts text (such as words, phrases, or entire paragraphs of text) into vectors in numerical form, enabling computers to process and understand natural language. The embedding model "text-embedding-ada-002" is widely used in text search, text similarity, and code search, and surpasses the previous most powerful embedding model Da Vinci in most tasks, while the price is 99.8% lower than Da Vinci.

[0057] Step S2: map construction;

[0058] Step S2.1: After retrieving relevant documents, the knowledge graph builder component is responsible for converting the retrieved relevant document content into a structured knowledge graph.

[0059] The knowledge graph builder component uses mistral local LLM for deep semantic analysis to identify key information in related documents, such as entities, attributes, and events. Through the named entity recognition (NER) function of mistral local LLM, entities such as names, places, organizations, etc. in related documents are extracted, the semantic relationships in related documents are analyzed, the attributes and relationships between entities are extracted, and the nodes and edges of the knowledge graph are constructed. In addition, the nomic-embed-text embedding model is used in combination to map entities, attributes, and relationships to a high-dimensional vector space for subsequent similarity comparison and expansion of the knowledge graph. Finally, the extracted nodes and edges are stored in the Neo4j graph database for easy query and update.

[0060] Step S2.2: extract keywords;

[0061] While creating the graph, the question parsing and keyword extractor component is a parallel component responsible for deeply understanding the question input by the user, identifying the subtasks in the question, generating the corresponding subquestions, extracting key information from it, and then converting (rewriting) the extracted key information into keywords. The question parsing and keyword extractor component uses dependency parsing (Dependency Parsing) and semantic role labeling (Semantic Role Labeling, SRL) technology to ensure that the extracted keywords can accurately reflect the core intention of the user input question.

[0062] In the process of extracting keywords, the question parsing and keyword extractor component first needs to perform semantic analysis on complex questions to identify the main entities and key relationships. Through semantic analysis, the core verbs, related entities and their relationships of the question can be clarified, so as to determine which parts of the question can be processed independently. This semantic analysis method usually relies on dependency parsing (Dependency Parsing) and semantic role labeling (Semantic Role Labeling, SRL). Among them, dependency parsing helps the system understand the grammatical relationship between words in a sentence, while semantic role labeling clarifies the semantic role of each entity in a sentence by identifying the predicate and its related arguments.

[0063] Once the subtasks in the problem are identified, the system will generate corresponding sub-problems. These sub-problems are expressed in natural language and can be solved independently. For example, given a complex question containing multiple entity relationships, such as "What are the main indications for coronary intervention in the treatment of heart disease? In addition, what are the risks and complications of coronary intervention?", dependency parsing first analyzes the grammatical structure of the sentence and identifies the various components in the sentence and their grammatical relationships.

[0064] For example, Figure 2 As shown in the figure, for the question "What are the main indications for coronary intervention in the treatment of heart disease? In addition, what are the risks and complications of coronary intervention?", dependency parsing will identify the following key components and relations:

[0065] Predicate: "what is", "what are there";

[0066] Subject: "Main indications for coronary intervention", "Risks and complications of coronary intervention";

[0067] Object: "Indications for coronary intervention", "Risks and complications";

[0068] Modifiers: "in the treatment of heart disease", "coronary intervention".

[0069] Through dependency parsing, the core verb (predicate) of the sentence and its related syntactic components (such as subject and object) can be identified. In this example, dependency parsing helps to decompose the original problem into two simpler sub-problems:

[0070] “What are the main indications for coronary intervention in the treatment of heart disease?”;

[0071] "What are the risks and complications of coronary intervention in the treatment of heart disease?".

[0072] When dealing with the two sub-questions "What are the main indications for coronary intervention in the treatment of heart disease?" and "What are the risks and complications of coronary intervention in the treatment of heart disease?", Semantic Role Labeling (SRL) labels the predicates and their related arguments in each sentence and assigns appropriate semantic roles to these arguments.

[0073] The following formulas further explain in detail how to handle the complex question "In the treatment of heart disease, what are the main indications for coronary intervention? In addition, what are the risks and complications of coronary intervention?" through dependency parsing (Dependency Parsing) and semantic role labeling (SRL).

[0074] Assume that the sentence S is "In the treatment of heart disease, what are the main indications for coronary intervention? In addition, what are the risks and complications of coronary intervention?" The goal of dependency parsing is to generate a dependency tree in which each word w i With its central word h i and dependency relationship i , the parsing task can be expressed as:

[0075]

[0076] Where T is the optimal dependency tree; τ(S) is the set of all possible dependency trees; Score(h i ,w i ,r i ) is the word w i As a word h i , and there is a dependency relationship between them. i score.

[0077] After dependency parsing, the system decomposes the problem into two sub-problems. In the processing of each sub-problem, semantic role labeling is used to identify the predicate and arguments of the sentence and assign semantic roles to these arguments. For example, for the sub-problem "What are the main indications for coronary intervention in the treatment of heart disease?", the goal of semantic role labeling is to label the corresponding semantic roles for the words in the sentence. Assume y t represents the semantic role label of the t-th word. The SRL model is expressed by the following formula:

[0078] h t =BiLSTM(x t ,h t-1 )

[0079] y t =softmax(Wh t +b)

[0080] Where: h t is the hidden state based on bidirectional LSTM; x t is the input word embedding or feature vector; y t is the semantic role label of the t-th word.

[0081] For the sub-question "Who are the actors in the movie ET?", SRL labels it as follows:

[0082] "Main indications for coronary intervention" (Theme);

[0083] ● “What is” (Predicate);

[0084] "In heart disease treatment" (Location);

[0085] Similarly, for the sub-question “Which actors from the movie ET are Argentinian?”, SRL will identify:

[0086] "Risks and complications of coronary intervention" (Theme);

[0087] · “What are there” (Predicate / predicate);

[0088] "In heart disease treatment" (Location);

[0089] These semantic roles help the system correctly understand the keywords of each sub-question and thus find accurate answers in the knowledge base.

[0090] Finally, for the first sub-question "What are the main indications for coronary intervention in the treatment of heart disease?", the keywords extracted were:

[0091] K1={heart disease, coronary intervention, main indication}

[0092] For the second sub-question, “What are the risks and complications of coronary intervention in the treatment of heart disease?”, the keywords extracted were:

[0093] K2 = {heart disease, coronary intervention, risk, complications}

[0094] Step S2.3: obtaining the associative prompt words;

[0095] For the extracted keywords, the present invention uses an external interface to call the large language model (GPT-4) for few-shot learning, and generates relevant associative prompts for these keywords through its powerful semantic understanding and association capabilities. The generated associative prompts not only expand the semantic scope of the keywords, but also help capture more relevant information when constructing Neo4j queries.

[0096] For K1 (the keyword of the first sub-question), the generated associative prompt word L1 is:

[0097] L1={coronary artery stenosis, coronary artery obstruction, angina pectoris, acute coronary syndrome, myocardial infarction}

[0098] For K2 (keyword of the second sub-question): the generated associative prompt word L2 is:

[0099] L2={vascular injury, stent thrombosis, bleeding, infection, myocardial infarction}

[0100] Step S2.4: Obtain the query command of the knowledge graph;

[0101] Next, the generated associative prompt words and sub-questions are combined to generate multiple query commands suitable for the Neo4j knowledge graph. Each query command expands the query scope through associative prompt words, thereby improving the comprehensiveness and accuracy of information retrieval.

[0102] For the first sub-question, the generated Neo4j query commands query the indication information related to coronary intervention from different perspectives (such as coronary intervention, indication, and association with heart disease treatment).

[0103] For the second sub-question, the generated Neo4j query command combines different associative prompt words to ensure that the qualified "risks and complications" information related to "coronary intervention" is retrieved from multiple channels.

[0104] Execute all generated Neo4j query commands to obtain subgraphs related to the subquestions from the knowledge graph:

[0105] G 1a G 1b G 1c =execute_Neo4j_queries(C 1a ,C 1b ,C 1c )

[0106] G 2a G 2b G 2c =execute_Neo4j_queries(C 2a ,C 2b ,C 2c )

[0107] By executing multiple query commands, the system retrieves relevant information from different perspectives to obtain the most comprehensive answer. 1a G 1b G 1c The results in G are used to identify all the actors in the movie "ET", and G 2a G 2b G 2c Then filter out actors whose nationality is Argentina.

[0108] Step S2.5: Mapping of knowledge graph to text and text compression;

[0109] After obtaining the sub-knowledge graph information G1 and G2 for the two sub-questions "What are the main indications for coronary intervention in the treatment of heart disease?" and "What are the risks and complications of coronary intervention in the treatment of heart disease?", it is necessary to convert these graph information into text form and compress and convert the redundant or repeated information in the text. This process involves mapping knowledge graphs to text and text compression technology.

[0110] For the process of converting knowledge graph to text, a template-based approach is adopted, that is, using predefined sentence templates, filling structured data into these templates to generate natural language text. The advantage of this method is that it is simple and controllable. Use specific templates such as "[treatment method] is suitable for [indication]" to fill in the node information in the knowledge graph. For example, for the question "In the treatment of heart disease, what are the main indications for coronary intervention?", the sentence generated by the template method may be: "Coronary intervention is suitable for relieving angina pectoris, acute coronary syndrome, etc.".

[0111] After obtaining the text information, the next step is to compress and transform the redundant or repeated information in the text. In terms of information compression, even if the retriever can retrieve relevant documents from a huge graph database, it still faces the challenge of processing a large amount of retrieval information. Some studies have attempted to solve this problem by expanding the context length of LLM, but the current LLM is still limited by the context. Therefore, it is particularly important to use information compression transformation to improve the expressiveness of LLM in this process by reducing information noise, solving the context length limitation and improving the generation effect.

[0112] First, we train an information compression transformer. In the compression phase, we input , generating a compressed sequence C extracted , this compressed sequence represents the concise information in the sub-knowledge graph. The purpose of training is to make the compressed sequence C extracted As close as possible to the actual context sequence C truth , the loss function used is divided into three parts:

[0113] 1. Matching loss L between generated sequence and target sequence match ;

[0114] 2. Information integrity loss L info ;

[0115] 3. Contextual coherence loss L context .

[0116] Specifically, for part 1, the matching loss L between the generated sequence and the target sequence is match Mainly used to measure the compression sequence C generated by the model extracted With the actual sequence C truth The degree of matching can usually be calculated by cross-entropy loss:

[0117]

[0118] Where T is the sequence length, K is the vocabulary size, and y t,k and They represent the probability distribution of the kth word in the target sequence and the generated sequence at time step t respectively.

[0119] For Part 2, the information integrity loss L info It aims to ensure that the compressed text retains important information in the original text and avoids missing key content. It can usually be measured by information entropy or information gain:

[0120]

[0121] Among them, H(C extracted |S input ) is a given input text S input The compressed sequence C output when extracted The information entropy of info It can ensure that the generated sequence retains the key information of the original input.

[0122] For Part 3, the contextual coherence loss L context It is mainly used to ensure that the compressed text is semantically and logically coherent. The similarity between the generated sequence and the context can usually be measured by using the latent layer features of a bidirectional language model (such as BERT):

[0123]

[0124] Among them, BERT (C extracted ) and BERT(S input ) are the BERT encoding representations of the generated sequence and the input text at time step t, respectively. Through the contextual coherence loss L context The generated sequences are encouraged to be contextually consistent with the original input.

[0125] The final comprehensive loss function L is the weighted sum of the above three loss terms:

[0126] L=α·L match +β·L info +γ·L context

[0127] Among them, α, β, and γ are weight coefficients used to adjust the contribution of the three loss terms to the overall loss. By adjusting these weights, the model can pay more attention to the matching degree between the text and the target sequence, the completeness of the information, and the coherence of the context during the compression process.

[0128] Step S3: Generate a natural language answer;

[0129] Finally, the compressed text and question are used as input and then fed into the answer generation component, which uses the GPT-4 large language model. The GPT-4 large language model understands the compressed text and question input and generates a natural language answer that is closely related to the input question content.

[0130] Specifically, the compressed text and question are first connected into an input sequence; the GPT-4 large language model processes this input sequence through its deep language understanding ability and generates a most likely answer sequence based on the context. During the answer generation process, the model considers the key information in the text and the specific requirements of the question to ensure that the generated answer is both accurate and coherent.

[0131] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating a large language model based on retrieval enhancement, characterized in that: The following steps are involved: Step S1: The document retrieval engine component retrieves document summaries similar to the input question based on the question, and uses text embedding technology to convert the input question and document summary into points in a high-dimensional vector space, and quickly locates relevant documents by calculating the similarity between these vectors; Step S2: The knowledge graph builder component converts the retrieved relevant documents into a structured knowledge graph; at the same time, the question parsing and keyword extractor component uses dependency syntax parsing and semantic role labeling technology to perform semantic analysis on the input question, identify the subtasks in the question, generate corresponding subquestions, and extract key information from the input question, and then convert it into keywords; call the large language model for few-sample learning for the keywords to generate related associative prompt words; combine the generated associative prompt words and subquestions to generate multiple query commands suitable for the knowledge graph; By executing multiple query commands, the sub-knowledge graph information of the sub-question is obtained, the sub-knowledge graph information is converted into text form, and the redundant or repeated information in the text is compressed and converted; Step S3: Input the compressed converted text and question into the GPT-4 large language model to generate a natural language answer that is closely related to the input question content.

2. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: In step S1, the file retrieval engine component adopts a small text block retrieval strategy during retrieval.

3. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: In step S1, the text embedding technology calls the embedding model text-embedding-ada-002.

4. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: In step S2, the knowledge graph builder component uses the mistral local LLM to perform deep semantic analysis, identify key information in relevant documents, extract entities in relevant documents through the named entity recognition function of the mistral local LLM, analyze the semantic relationships in relevant documents, extract attributes and relationships between entities, and construct nodes and edges of the knowledge graph; use the nomic-embed-text embedding model to map entities, attributes and relationships to a high-dimensional vector space; and store the extracted nodes and edges in the Neo4j graph database.

5. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: The dependency parsing is used to identify the core verbs of a sentence and their related syntactic components; the goal of the dependency parsing is to generate a dependency tree, where each word w i With its central word h i and dependency relationship i , the parsing task is expressed as: Where T is the optimal dependency tree; τ(S) is the set of all possible dependency trees; Score(h i ,w i ,r i ) is the word w i As a word h i , and there is a dependency relationship between them. i score.

6. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: The semantic role labeling is used to identify the predicate and arguments of a sentence and assign semantic roles to these arguments; assuming y t represents the semantic role label of the t-th word. The SRL model is expressed by the following formula: h t =BiLSTM(x t ,h t-1 ) y t =softmax(Wh t +b) Where: h t is the hidden state based on bidirectional LSTM; x t is the input word embedding or feature vector; y t is the semantic role label of the t-th word.

7. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: In step S2, a template-based method is used to convert the sub-knowledge graph information into text form, that is, predefined sentence templates are used and structured data are filled into these templates to generate natural language text.

8. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: In step S2, the specific implementation steps of the compression conversion are as follows: Train an information compression transformer. In the compression phase, given the input text S input , generating a compressed sequence C extracted The purpose of training is to make the compressed sequence C extracted As close as possible to the actual context sequence C truth , the loss function used is: the matching loss L between the generated sequence and the target sequence match , information integrity loss L info and contextual coherence loss L context ; The matching loss L between the generated sequence and the target sequence match Used to measure the compressed sequence C generated by the model extracted With the actual sequence C truth The matching degree, the information integrity loss L info It is used to ensure that the compressed text retains important information in the original text and avoids missing key content. The context coherence loss L context Used to ensure that the compressed text is semantically and logically coherent; The final comprehensive loss function L is the weighted sum of the above three loss terms: L=α·L match +β·L info +γ·L context Among them, α, β, and γ are weight coefficients used to adjust the contribution of the three loss terms to the overall loss.

9. The method for generating a large language model based on retrieval enhancement according to claim 8, characterized in that: The matching loss L between the generated sequence and the target sequence match Calculated by cross entropy loss; the information integrity loss L info Measured by information entropy or information gain; the context coherence loss L context The similarity between the generated sequence and the context is measured by using the latent layer features of the bidirectional language model.

10. The method for generating a large language model based on retrieval enhancement according to claim 1, characterized in that: In step S3, the compressed text and question are first connected into an input sequence. The GPT-4 large language model processes this input sequence through its deep language understanding ability and generates a most likely answer sequence based on the context.

Images