Large language model question and answer method and system based on retrieval enhancement generation scheme

Abstract

The invention discloses a large language model question and answer method based on a retrieval enhancement generation scheme. The method comprises the following steps: segmenting a source document into a plurality of text blocks; extracting element instances from the text blocks; converting the element instance into a readable element abstract; constructing the element abstract into a graph community based on a graph theory scheme; generating a community-level abstract for the graph community; generating community answers according to the digests of the community levels and reducing the community answers to obtain global answers, and completing large language model questions and answers based on the retrieval enhancement generation scheme. The invention further discloses a system for realizing the retrieval enhancement generation scheme-based large language model question and answer method. According to the large language model question and answer method and system based on the retrieval enhancement generation scheme, entity knowledge graph and community abstract generation technologies are combined, question and answer of the large language model are achieved based on the retrieval enhancement generation scheme, reliability is higher, and accuracy is better.

Description

Technical Field

[0001] The present invention belongs to the field of electrical automation, and in particular relates to a large language model question answering method and system based on a retrieval enhancement generation scheme. Background Art

[0002] With the development of economy and technology and the improvement of people's living standards, electricity has become an indispensable secondary energy source in people's production and life, bringing endless convenience to people's production and life. Therefore, ensuring the stable and reliable supply of electricity has become one of the most important tasks of the power system.

[0003] At present, with the rapid development of intelligent power systems, large language models have played a huge role in the service field and knowledge query field of power systems, greatly improving the operation efficiency and reliability of power systems.

[0004] Large language models have achieved certain results in related fields of natural language processing. However, the knowledge of large language models comes entirely from their training data; therefore, when these models are applied to process private texts or document collections that have never been seen in model training, large language models alone are often unable to efficiently retrieve relevant information from large amounts of text. To overcome this problem, researchers proposed a retrieval-augmented generation (RAG) scheme that combines information retrieval and text generation. By retrieving relevant information from external knowledge sources, the generation capabilities of large language models are enhanced, enabling them to answer questions about specific document collections.

[0005] Although the retrieval-enhanced generation method performs well in document question-answering tasks, it also has significant limitations when dealing with global problems. For example, when answering query-focused summarization (QFS) task questions, it requires a global understanding of the entire document corpus, while traditional retrieval-enhanced generation methods focus more on accurate retrieval of specific query content, so it is difficult to effectively deal with such tasks. Therefore, the existing large language model question-answering solutions based on retrieval-enhanced generation methods still have the problems of low reliability and poor accuracy. Summary of the invention

[0006] One of the purposes of the present invention is to provide a large language model question answering method based on a retrieval enhanced generation scheme with high reliability and good accuracy.

[0007] A second objective of the present invention is to provide a system for implementing the large language model question answering method based on the retrieval enhanced generation scheme.

[0008] The large language model question answering method based on the retrieval enhancement generation scheme provided by the present invention comprises the following steps:

[0009] S1. Divide the source document into several text blocks;

[0010] S2. extracting element instances from the text block obtained in step S1;

[0011] S3. Convert the element instance obtained in step S2 into a readable element summary;

[0012] S4. Based on a graph theory solution, construct the element summary obtained in step S3 into a graph community;

[0013] S5. Generate a community-level summary for the graph community obtained in step S4;

[0014] S6. Generate a community answer based on the community-level summary obtained in step S5, and reduce it to obtain a global answer to complete the large language model question and answer based on the retrieval-enhanced generation scheme.

[0015] The step S1 of dividing the source document into a plurality of text blocks specifically includes the following steps:

[0016] Extract text from the source document and divide the extracted text into several text blocks according to the processing window size of the large language model.

[0017] The method of dividing the extracted text into a plurality of text blocks according to the processing window size of the large language model specifically includes the following steps:

[0018] Physical Integrity Assessment:

[0019] After each round of extraction, the Logit bias of the large language model is adjusted to ensure that the large language model only returns two judgments, "yes" or "no", and determines whether there are missing entities: if the large language model returns "yes", it means that there are no missing entities; if the large language model returns "no", it proceeds to the subsequent steps to perform entity supplementary extraction;

[0020] Entity Supplement Extraction:

[0021] When the large language model detects entity omission, the large language model is strengthened to pay attention to the missing information by enhancing the guiding prompts;

[0022] Process Optimization:

[0023] The following formula is used to evaluate the improvement of each round of extraction:

[0024]

[0025] Where E nis the improvement rate of the nth round of extraction; M n is the number of entities extracted in round n; M n-1 is the number of entities extracted in round n-1;

[0026] For E n The value of E is monitored: n If it is less than the set threshold, the extraction is deemed complete and the extraction process is terminated early.

[0027] Step S2, extracting element instances from the text block obtained in step S1, specifically includes the following steps:

[0028] For each text block, a large language model is used to identify and extract nodes and edges;

[0029] Extract all entities in the text block; the entities include entity name, entity type and entity description;

[0030] Identify all relationships between entities; the relationships include source entities, target entities, and corresponding relationship descriptions.

[0031] Step S3 of converting the element instance obtained in step S2 into a readable element summary specifically includes the following steps:

[0032] Similarity calculation and clustering:

[0033] The following formula is used to calculate the semantic similarity of element instances:

[0034]

[0035] Where SemanticSimilarity(A,B) is the semantic similarity value between element instance A and element instance B; V A is the semantic vector of element instance A; V B is the semantic vector of element instance B; V A and V B All are generated by deep language models;

[0036] Using a clustering algorithm, element instances with similarity greater than a set value are grouped together to form an element candidate group;

[0037] Merge instance information:

[0038] Filter the element instance information in each group of element candidates, extract high-frequency information and eliminate redundancy to obtain a core information pool;

[0039] Abstractive summarization using large language models:

[0040] Convert the content in the core information pool into a natural language description, expressed as:

[0041] S element =LLM({I 1 ,I 2 ,...,I n})

[0042] Where S element is the generated element summary; 1 ,I 2 ,...,I n} is the instance group in the core information pool;

[0043] Formatted output:

[0044] Unify the output format and output.

[0045] The graph-theoretic solution described in step S4 constructs the element summary obtained in step S3 into a graph community, which specifically includes the following steps:

[0046] A. Local improvements:

[0047] Treat each node as a separate community; move each node to an adjacent community and determine: if the value of the community's modularity increase after moving the node to the adjacent community is greater than the set threshold, then move the node to the adjacent community;

[0048] Repeat step A until moving any node cannot increase the modularity of the community by a value greater than the set threshold;

[0049] The modularity Q is calculated using the following formula:

[0050]

[0051] Where m is the total number of edges in the network; A ij is the element in the i-th row and j-th column of the adjacency matrix A. If there is an edge connection between node i and node j, then A ij =1, if there is no edge connection between node i and node j, then A ij =0; k i is the degree of node i; k j is the degree of node j; δ(c i ,c j ) is the Kronecker delta function, c i is the community of node i, c j is the community of node j, if c i =c j Then δ(c i ,c j )=1, if c i ≠c j Then δ(c i ,cj )=0;

[0052] B. Graph Compression:

[0053] Communities are considered as supernodes and compressed; after compression, each supernode represents a community in the previous stage, and the edges between supernodes represent the connections between communities in the graph;

[0054] C. Secondary Optimization:

[0055] Community refinement:

[0056] First, perform subcommunity division: for each initially generated community C, re-divide the internal nodes of community C into several subcommunities, and each node maintains direct connection with the nodes in the subcommunity whose number is greater than the set value;

[0057] Screening of strongly connected subcommunities: using the strongly connected component method to ensure that the nodes in each subcommunity are connected through paths; if the internal connectivity of a subcommunity is lower than the set threshold, the subcommunity is split or merged with other subcommunities until the connectivity requirements are met;

[0058] Subcommunity reorganization and merger:

[0059] For subcommunities, similarity thresholds are used to merge them; the similarity measurement standard is the edge weight ratio between subcommunities, expressed as:

[0060]

[0061] In the formula, Similarity(C i ,C j ) is subcommunity C i With C j The edge weight ratio between i,j Subcommunity C i With C j The sum of the edge weights between i,i Subcommunity C i The sum of internal edge weights; W j,j Subcommunity C j The sum of internal edge weights;

[0062] Reassign isolated nodes or weakly connected nodes to the nearest strongly connected subcommunity;

[0063] Super Node Aggregation:

[0064] A new network is constructed, where each finalized community is considered as a “supernode”; the edge weight between two supernodes is the sum of the edge weights between the two communities in the original network, expressed as:

[0065]

[0066] Where W′(C i ,C j ) is the edge weight between super nodes in the new network; W(u,v) is the edge weight between nodes u and v in the original network;

[0067] Build a simplified network, form a simplified network with new super nodes and corresponding edges, and re-optimize the modularity based on the network;

[0068] D. Repeat steps A to C until the transformation value of the modularity after two consecutive repetitions is less than the set threshold, and finally construct the element summary as a graph community.

[0069] Step S5 generates a community-level summary for the graph community obtained in step S4, specifically including the following steps:

[0070] For leaf-level communities: add the element summary of the leaf-level community to the processing window of the large language model until the context token limit of the processing window is reached; the element summary includes nodes, edges, and covariates; for each leaf-level community, sort the leaf-level communities in descending order based on the sum of the degrees of the source and target nodes as the priority, and add the description of the source node, target node, linked covariates, and the edge itself;

[0071] For high-level communities: If the summaries of all elements meet the token limit of the processing window of the large language model, the element summaries of the high-level community are added to the processing window of the large language model until the context token limit of the processing window is reached; the element summaries include nodes, edges, and covariates; for each high-level community, the high-level communities are sorted in descending order with the sum of the degrees of the source and target nodes as the priority, and the descriptions of the source nodes, target nodes, linked covariates, and edges themselves are added;

[0072] If the summary of an element does not meet the token limit of the processing window of the large language model, iteratively sort the subcommunities of the high-level community in descending order according to the number of tokens in the element summary, and use the subcommunity summary to replace the corresponding element summary that does not meet the token limit of the processing window of the large language model until it meets the token limit of the processing window of the large language model.

[0073] Step S6 generates a community answer based on the community-level summary obtained in step S5, and reduces it to obtain a global answer, which specifically includes the following steps:

[0074] Get questions from users of the large language model;

[0075] Based on the questions raised by users, several community-level summaries generate several community answers in parallel;

[0076] The large language model scores the community answers obtained and deletes the community answers with scores lower than the set value;

[0077] For the retained community answers, the relevance scores of the retained community answers are calculated, and the best answers are added to the processing window of the large language model until the token limit of the processing window of the large language model is reached to generate the final global answer.

[0078] The present invention also provides a system for implementing the large language model question-answering method based on the retrieval enhancement generation scheme, including a document segmentation module, an instance extraction module, an instance conversion module, a graph community construction module, a summary generation module and a question-answering module; the document segmentation module, the instance extraction module, the instance conversion module, the graph community construction module, the summary generation module and the question-answering module are connected in series in sequence; the document segmentation module is used to segment the source document into a plurality of text blocks, and upload the data information to the instance extraction module; the instance extraction module is used to extract element instances from the obtained text blocks according to the received data information, and upload the data information to the instance conversion module; the instance conversion module is used to According to the received data information, the obtained element instance is converted into a readable element summary, and the data information is uploaded to the graph community construction module; the graph community construction module is used to construct the obtained element summary into a graph community based on the received data information and the graph theory scheme, and upload the data information to the summary generation module; the summary generation module is used to generate a community-level summary for the obtained graph community based on the received data information, and upload the data information to the question and answer module; the question and answer module is used to generate a community answer based on the received data information and the obtained community-level summary, and reduce it to a global answer, so as to complete the large language model question and answer based on the retrieval enhancement generation scheme.

[0079] The large language model question and answer method and system based on the retrieval enhancement generation scheme provided by the present invention combines the entity knowledge graph and the community summary generation technology. Based on the retrieval enhancement generation scheme, it not only realizes the question and answer of the large language model, but also has higher reliability and better accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 The figure is a schematic diagram of the method flow of the present invention.

[0081] Figure 2 Schematic diagram of the functional modules of the system of the present invention. DETAILED DESCRIPTION

[0082] like Figure 1 The method flow diagram of the method of the present invention is shown as follows: the large language model question answering method based on the retrieval enhancement generation scheme disclosed in the present invention comprises the following steps:

[0083] S1. Segment the source document into several text blocks; specifically, the steps include:

[0084] Extract text from the source document and divide the extracted text into several text blocks according to the processing window size of the large language model. The reason for block processing is to ensure that each text block can be processed within the context window of the LLM (large language model) and entities and relations can be extracted efficiently. If the text block is too large, the context window of the LLM may cause the extraction accuracy to decrease, while if the text block is too small, the processing efficiency may decrease. Therefore, it is necessary to balance the processing efficiency and extraction accuracy.

[0085] The specific steps include:

[0086] In the process of document entity extraction, in order to balance efficiency and quality, a "multi-round picking" mechanism is adopted. By limiting the maximum number of rounds, the large language model is encouraged to gradually detect and supplement entities that may be missed in the previous round in each round of extraction. This multi-stage process ensures the comprehensiveness of the extraction and avoids information loss or noise introduction due to large text blocks. The following are the specific steps of the "multi-round picking" mechanism:

[0087] Physical Integrity Assessment:

[0088] After each round of extraction, the Logit Bias of the large language model is adjusted to ensure that the large language model only returns two judgments, "yes" or "no", and determines whether there are missing entities: if the large language model returns "yes", it means that there are no missing entities; if the large language model returns "no", it proceeds to the subsequent steps to perform entity supplementary extraction;

[0089] Entity Supplement Extraction:

[0090] When the large language model detects entity omissions, it strengthens the large language model's attention to the missing information by enhancing guiding prompts; guiding prompts such as "many entities were missed in the last extraction"; this statement emphasizes the severity of the omissions and prompts the model to increase the search for the missing entities in the next round of extraction;

[0091] Since the model can automatically fill in the gaps in multiple rounds of picking, larger text blocks can be used, thus improving overall processing efficiency;

[0092] Process Optimization

[0093] In order to further optimize the efficiency and quality of multiple rounds of picking, the following formula is introduced to evaluate the improvement of each round of extraction:

[0094]

[0095] Where En is the improvement rate of the nth round of extraction; M n is the number of entities extracted in round n; M n-1 is the number of entities extracted in round n-1;

[0096] For E n The value of E is monitored: n If it is less than the set threshold (e.g. 5%), the extraction is deemed complete and the extraction process is terminated early to save computing resources;

[0097] Through this step, the document is divided into multiple small segments with punctuation marks as markers, and low-density important entities are prioritized in each round, and information is gradually extracted, which reduces the alignment requirements of the input data; this step can use larger text blocks without reducing quality or forcing the introduction of noise;

[0098] In specific implementation, for example, all common problems and error information can be extracted from the relevant documents of the ERP financial module and divided into multiple text blocks; these text blocks should contain key information such as system problems, solutions, error codes, transaction codes, etc.; the block size should be adjusted according to the context window size of the LLM to ensure that key context is not lost during text analysis; for example, the description of the "customer clearing error processing" problem can be divided into small text segments containing error reasons, error prompts, solutions, and transaction codes to facilitate subsequent entity and relationship extraction;

[0099] S2. Extracting element instances from the text block obtained in step S1; specifically comprising the following steps:

[0100] For each text block, a large language model is used to identify and extract nodes and edges;

[0101] Extract all entities in the text block; the entities include entity name, entity type and entity description;

[0102] Identify all relationships between entities; the relationships include source entities, target entities, and corresponding relationship descriptions;

[0103] In addition, covariate prompts can be used, that is, secondary extraction prompts can be given for the types of entity nodes and the associations between entity nodes, providing additional information to enrich the description of nodes and edges in the graph and enhance the understanding and reasoning ability of the model;

[0104] In the specific implementation, various entities in the financial module (such as error codes, transaction codes, accounts, suppliers, customers, WBS elements, etc.) are extracted and described, and the relationships between entities are identified, such as the association between error causes and solutions, the correspondence between transaction codes and operations, etc. When extracting these entities and relationships, LLM will refer to the professional terms and operation logic related to the financial module to ensure that the extraction of entities and relationships is accurate and comprehensive;

[0105] S3. Convert the element instance obtained in step S2 into a readable element summary; specifically comprising the following steps:

[0106] After extracting entities and relationships between entities, LLM further converts this information into a readable summary. The entities, relationships, and descriptions extracted by LLM are actually simplifications and summaries of the original text. However, when relying on LLM to create an independent and complete concept summary, it may contain some content that is not explicitly stated in the original text, such as implicit relationships between entities. In other words, LLM may infer some relationships that are not directly stated based on the context. In order to convert all such instance-level summaries into a single descriptive text block for each graph element (i.e., entity nodes, relationship edges, and declared covariates), further LLM summaries are required for the matching instance groups.

[0107] Similarity calculation and clustering:

[0108] The following formula is used to calculate the semantic similarity of element instances:

[0109]

[0110] Where SemanticSimilarity(A,B) is the semantic similarity value between element instance A and element instance B; V A is the semantic vector of element instance A; V B is the semantic vector of element instance B; V A and V B Both are generated by deep language models; semantic similarity is used to capture deep semantic information and is suitable for comparison of long texts and complex relationships;

[0111] Using a clustering algorithm, grouping element instances whose similarity is greater than a set value into a group to form an element candidate group; the clustering algorithm includes a density clustering algorithm (such as DBSCAN) or a hierarchical clustering algorithm;

[0112] Merge instance information:

[0113] Filter the element instance information in each group of element candidates, extract high-frequency information and eliminate redundancy to obtain a core information pool;

[0114] Abstractive summarization using large language models:

[0115] Convert the content in the core information pool into a natural language description, expressed as:

[0116] S element =LLM({I 1 ,I 2 ,...,I n})

[0117] Where S element is the generated element summary; 1 ,I 2 ,...,I n} is the instance group in the core information pool;

[0118] Formatted output:

[0119] The output format is unified and output; the output format includes entity name, category, attribute, relationship description, etc. The entity and its associated relationship will be converted into concise and complete descriptive text, which is consistent with the overall goal and query requirements of the large language model; each element in the graph will have a summarized description, and an index of the graph will be established to better find and reference these elements;

[0120] Through this step, entities and their associated relationships are converted into concise and complete descriptive text that is consistent with the overall goals and query requirements of LLM; each element in the diagram will have a summarized description, and an index of the diagram will be established to better find and reference these elements;

[0121] In specific implementation, for example, for a problem of "supplier clearing failure", a summary of the error cause, associated transaction code, and solution is generated. LLM's generation work in this step is not only based on direct information in the document, but also infers certain implicit relationships, such as the connection between the error code and common operational errors.

[0122] S4. Based on the graph theory solution, the element summary obtained in step S3 is constructed into a graph community; specifically, the steps include:

[0123] The index created in the previous step can be modeled as a homogeneous undirected weighted graph, in which entity nodes are connected by relationship edges, and the edge weights represent the normalized counts of detected relationship instances; the scheme of the present invention adopts the following steps to divide the graph into communities, which can effectively restore the hierarchical community structure of large graphs. Each level of this hierarchical structure provides a community partition, which covers the nodes of the graph in a mutually exclusive and set-exhaustive manner, thereby achieving a divide-and-conquer global summary;

[0124] A. Local improvements:

[0125] Treat each node as a separate community; move each node to an adjacent community and determine: if the value of the community's modularity increase after moving the node to the adjacent community is greater than the set threshold, then move the node to the adjacent community;

[0126] Repeat step A until moving any node cannot increase the modularity of the community by a value greater than the set threshold;

[0127] Modularity is an indicator to measure the quality of community division in the network. It evaluates whether the connection density of nodes within the community is higher than that of random connection. The higher the modularity value, the better the community division effect, that is, the connection between nodes within the community is tighter and the connection between communities is less. The modularity Q is calculated using the following formula:

[0128]

[0129] Where m is the total number of edges in the network; A ij is the element in the i-th row and j-th column of the adjacency matrix A. If there is an edge connection between node i and node j, then A ij =1, if there is no edge connection between node i and node j, then A ij =0; k i is the degree of node i; k j is the degree of node j; δ(c i ,c j ) is the Kronecker delta function, c i is the community of node i, c j is the community of node j, if c i =c j Then δ(c i ,c j )=1, if c i ≠c j Then δ(c i ,c j )=0;

[0130] B. Graph Compression:

[0131] Communities are considered as supernodes and compressed; after compression, each supernode represents a community in the previous stage, and the edges between supernodes represent the connections between communities in the graph;

[0132] C. Secondary Optimization:

[0133] This step is used to repair some areas within the community that may not be fully optimized, ensuring that the generated community is more detailed and accurate; this step finds disconnected parts within each community (that is, there may be some nodes that are not directly connected to each other), and then it further optimizes the division results by moving these disconnected parts to more appropriate communities; specifically, it includes the following steps:

[0134] Community refinement:

[0135] First, perform subcommunity division: for each initially generated community C, further refine it and re-divide the internal nodes of community C into several highly connected subcommunities; each node maintains a direct connection with the number of nodes in its subcommunity that is greater than the set value, which is defined as "internal connectivity"; if the edge weights between a node and multiple subcommunities are similar, then select the subcommunity with the largest sum of its edge weights;

[0136] Screening of strongly connected subcommunities: using the strongly connected component method to ensure that the nodes in each subcommunity are connected through paths; if the internal connectivity of a subcommunity is lower than the set threshold, the subcommunity is split or merged with other subcommunities until the connectivity requirements are met;

[0137] Subcommunity reorganization and merger:

[0138] For subcommunities, similarity thresholds are used to merge them; the similarity measurement standard is the edge weight ratio between subcommunities, expressed as:

[0139]

[0140] In the formula, Similarity(C i ,C j ) is subcommunity C i With C j The edge weight ratio between i,j Subcommunity C i With C j The sum of the edge weights between i,i Subcommunity C i The sum of internal edge weights; W j,j Subcommunity C j The sum of internal edge weights;

[0141] Reassign isolated nodes or weakly connected nodes to the nearest strongly connected subcommunity to avoid the generation of isolated nodes or small fragmented communities;

[0142] Super Node Aggregation:

[0143] A new network is constructed, where each finalized community is considered as a “supernode”; the edge weight between two supernodes is the sum of the edge weights between the two communities in the original network, expressed as:

[0144]

[0145] Where W′(C i ,C j ) is the edge weight between super nodes in the new network; W(u,v) is the edge weight between nodes u and v in the original network;

[0146] Build a simplified network, form a simplified network with new super nodes and corresponding edges, and re-optimize the modularity based on the network;

[0147] D. Repeat steps A to C until the transformation value of the modularity after two consecutive repetitions is less than the set threshold, and finally construct the element summary as a graph community;

[0148] In the specific implementation, a knowledge graph of financial error information is created through graph databases such as Neo4j; each node represents an entity in the error information, and the edge represents the relationship between different entities; the Py2neo library is used to develop the graph data storage function, batch read and merge nodes, automatically ignore duplicate data, and complete the efficient knowledge graph construction. Subsequently, the nodes are grouped to reflect the problem associations and common error propagation paths between different financial modules; the community division is optimized by maximizing the modularity to ensure that the connection within each community is tight and the connection between communities is loose;

[0149] S5. Generate a community-level summary for the graph community obtained in step S4; specifically, the steps include:

[0150] This step uses a method that can be extended to very large datasets to create report-like community summaries for each community. These summaries not only help understand the global structure and semantics of the dataset, but also provide valuable information to users independently. For example, users can find general themes of the overview data, discover report structures, and provide more details for each subtopic in the community summary divided at a certain level.

[0151] For leaf-level communities: add the element summary of the leaf-level community to the processing window of the large language model until the context token limit of the processing window is reached; the element summary includes nodes, edges, and covariates; for each leaf-level community, sort the leaf-level communities in descending order based on the sum of the degrees of the source and target nodes as the priority, and add the description of the source node, target node, linked covariates, and the edge itself;

[0152] For high-level communities: If the summaries of all elements meet the token limit of the processing window of the large language model, the element summaries of the high-level community are added to the processing window of the large language model until the context token limit of the processing window is reached; the element summaries include nodes, edges, and covariates; for each high-level community, the high-level communities are sorted in descending order with the sum of the degrees of the source and target nodes as the priority, and the descriptions of the source nodes, target nodes, linked covariates, and edges themselves are added;

[0153] If the summary of an element does not meet the token limit of the processing window of the large language model, iteratively sort the subcommunities of the high-level community in descending order according to the number of tokens in the element summary, and use the subcommunity summary to replace the corresponding element summary that does not meet the token limit of the processing window of the large language model until it meets the token limit of the processing window of the large language model;

[0154] In specific implementation, for example, when dealing with the "A / R Clearing Error" community, the summary will mention the relevant error information, transaction code and common solutions; for larger communities or upper-level communities, LLM will integrate the underlying communities to generate a more concise high-level summary to outline the overall problem status in the financial module; these summaries can help users understand the functions and common error information of each financial module;

[0155] S6. Generate a community answer based on the community-level summary obtained in step S5, and reduce it to obtain a global answer, completing the large language model question answering based on the retrieval enhancement generation scheme; specifically, the steps include:

[0156] Get questions from users of the large language model;

[0157] Based on the questions raised by users, several community-level summaries generate several community answers in parallel;

[0158] The large language model scores the community answers obtained and deletes the community answers with scores lower than the set value;

[0159] For the retained community answers, calculate the relevance scores of the retained community answers, and add the best answers to the processing window of the large language model until the token limit of the processing window of the large language model is reached to generate the final global answer;

[0160] In specific implementation, for example, when a user raises a financial error query, the system first processes the summaries of each community to generate answers for each community; LLM will score each answer (such as 0-100) to indicate the contribution of the answer to solving the query; answers with low scores will be filtered out, and only answers with high scores will enter the final integration stage; by integrating answers from multiple communities, a comprehensive error diagnosis conclusion for the entire ERP financial module is formed.

[0161] like Figure 2 The figure shows a schematic diagram of the functional modules of the system of the present invention: the system disclosed in the present invention for implementing the large language model question-answering method based on the retrieval enhancement generation scheme comprises a document segmentation module, an instance extraction module, an instance conversion module, a graph community construction module, a summary generation module and a question-answering module; the document segmentation module, the instance extraction module, the instance conversion module, the graph community construction module, the summary generation module and the question-answering module are connected in series in sequence; the document segmentation module is used to segment the source document into a plurality of text blocks, and upload the data information to the instance extraction module; the instance extraction module is used to extract element instances from the obtained text blocks according to the received data information, and upload the data information to the instance conversion module; The instance conversion module is used to convert the obtained element instance into a readable element summary according to the received data information, and upload the data information to the graph community construction module; the graph community construction module is used to construct the obtained element summary into a graph community according to the received data information and based on the graph theory scheme, and upload the data information to the summary generation module; the summary generation module is used to generate a community-level summary for the obtained graph community according to the received data information, and upload the data information to the question and answer module; the question and answer module is used to generate a community answer according to the received data information and the obtained community-level summary, and reduce it to a global answer, so as to complete the large language model question and answer based on the retrieval enhancement generation scheme.

Claims

1. A large language model question answering method based on a retrieval-enhanced generation scheme, comprising the following steps: S1. Divide the source document into several text blocks; S2. extracting element instances from the text block obtained in step S1; S3. Convert the element instance obtained in step S2 into a readable element summary; S4. Based on a graph theory solution, construct the element summary obtained in step S3 into a graph community; S5. Generate a community-level summary for the graph community obtained in step S4; S6. Generate a community answer based on the community-level summary obtained in step S5, and reduce it to obtain a global answer to complete the large language model question and answer based on the retrieval-enhanced generation scheme.

2. The large language model question answering method based on retrieval enhancement generation scheme according to claim 1 is characterized in that The step S1 of dividing the source document into a plurality of text blocks specifically includes the following steps: Extract text from the source document and divide the extracted text into several text blocks according to the processing window size of the large language model.

3. The large language model question answering method based on retrieval enhancement generation scheme according to claim 2 is characterized in that The method of dividing the extracted text into a plurality of text blocks according to the processing window size of the large language model specifically includes the following steps: Physical Integrity Assessment: After each round of extraction, the Logit bias of the large language model is adjusted to ensure that the large language model only returns two judgments, "yes" or "no", and determines whether there are missing entities: if the large language model returns "yes", it means that there are no missing entities; if the large language model returns "no", it proceeds to the subsequent steps to perform entity supplementary extraction; Entity Supplement Extraction: When the large language model detects entity omission, the large language model is strengthened to pay attention to the missing information by enhancing the guiding prompts; Process Optimization: The following formula is used to evaluate the improvement of each round of extraction: Where E n is the improvement rate of the nth round of extraction; M n is the number of entities extracted in round n; M n-1 is the number of entities extracted in round n-1; For E n The value of E is monitored: n If it is less than the set threshold, the extraction is deemed complete and the extraction process is terminated early.

4. The large language model question answering method based on retrieval enhancement generation scheme according to claim 3 is characterized in that Step S2, extracting element instances from the text block obtained in step S1, specifically includes the following steps: For each text block, a large language model is used to identify and extract nodes and edges; Extract all entities in the text block; the entities include entity name, entity type and entity description; Identify all relationships between entities; the relationships include source entities, target entities, and corresponding relationship descriptions.

5. The large language model question answering method based on retrieval enhancement generation scheme according to claim 4 is characterized in that Step S3 of converting the element instance obtained in step S2 into a readable element summary specifically includes the following steps: Similarity calculation and clustering: The following formula is used to calculate the semantic similarity of element instances: Where SemanticSimilarity(A,B) is the semantic similarity value between element instance A and element instance B; V A is the semantic vector of element instance A; V B is the semantic vector of element instance B; V A and V B All are generated by deep language models; Using a clustering algorithm, element instances with similarity greater than a set value are grouped together to form an element candidate group; Merge instance information: Filter the element instance information in each group of element candidates, extract high-frequency information and eliminate redundancy to obtain a core information pool; Abstractive summarization using large language models: Convert the content in the core information pool into a natural language description, expressed as: S element =LLM({I1,I2,...,I n }) Where S element is the generated element summary; {I1,I2,...,I n } is the instance group in the core information pool; Formatted output: Unify the output format and output.

6. The large language model question answering method based on retrieval enhancement generation scheme according to claim 5 is characterized in that The graph-theoretic solution described in step S4 constructs the element summary obtained in step S3 into a graph community, which specifically includes the following steps: A. Local improvements: Treat each node as a separate community; move each node to an adjacent community and determine: if the value of the community's modularity increase after moving the node to the adjacent community is greater than the set threshold, then move the node to the adjacent community; Repeat step A until moving any node cannot increase the modularity of the community by a value greater than the set threshold; The modularity Q is calculated using the following formula: Where m is the total number of edges in the network; A ij is the element in the i-th row and j-th column of the adjacency matrix A. If there is an edge connection between node i and node j, then A ij =1, if there is no edge connection between node i and node j, then A ij =0; k i is the degree of node i; k j is the degree of node j; δ(c i ,c j ) is the Kronecker delta function, c i is the community of node i, c j is the community of node j, if c i =c j Then δ(c i ,c j )=1, if c i ≠c j Then δ(c i ,c j )=0; B. Graph Compression: Communities are considered as supernodes and compressed; after compression, each supernode represents a community in the previous stage, and the edges between supernodes represent the connections between communities in the graph; C. Secondary Optimization: Community refinement: First, perform subcommunity division: for each initially generated community C, re-divide the internal nodes of community C into several subcommunities, and each node maintains direct connection with the nodes in the subcommunity whose number is greater than the set value; Screening of strongly connected subcommunities: using the strongly connected component method to ensure that the nodes in each subcommunity are connected through paths; if the internal connectivity of a subcommunity is lower than the set threshold, the subcommunity is split or merged with other subcommunities until the connectivity requirements are met; Subcommunity reorganization and merger: For subcommunities, similarity thresholds are used to merge them; the similarity measurement standard is the edge weight ratio between subcommunities, expressed as: In the formula, Similarity(C i ,C j ) is subcommunity C i With C j The edge weight ratio between i,j Subcommunity C i With C j The sum of the edge weights between ; W i,i Subcommunity C i The sum of internal edge weights; W j,j Subcommunity C j The sum of internal edge weights; Reassign isolated nodes or weakly connected nodes to the nearest strongly connected subcommunity; Super Node Aggregation: Build a new network that considers each finalized community as a "supernode"; The edge weight between two super nodes is the sum of the edge weights between the two communities in the original network, expressed as: Where W′(C i ,C j ) is the edge weight between super nodes in the new network; W(u,v) is the edge weight between nodes u and v in the original network; Build a simplified network, form a simplified network with new super nodes and corresponding edges, and re-optimize the modularity based on the network; D. Repeat steps A to C until the transformation value of the modularity after two consecutive repetitions is less than the set threshold, and finally construct the element summary as a graph community.

7. The large language model question answering method based on retrieval enhancement generation scheme according to claim 6 is characterized in that Step S5 generates a community-level summary for the graph community obtained in step S4, specifically including the following steps: For leaf-level communities: add the element summary of the leaf-level community to the processing window of the large language model until the context token limit of the processing window is reached; the element summary includes nodes, edges, and covariates; for each leaf-level community, sort the leaf-level communities in descending order based on the sum of the degrees of the source and target nodes as the priority, and add the description of the source node, target node, linked covariates, and the edge itself; For high-level communities: If the summaries of all elements meet the token limit of the processing window of the large language model, the element summaries of the high-level community are added to the processing window of the large language model until the context token limit of the processing window is reached; the element summaries include nodes, edges, and covariates; for each high-level community, the high-level communities are sorted in descending order with the sum of the degrees of the source and target nodes as the priority, and the descriptions of the source nodes, target nodes, linked covariates, and edges themselves are added; If the summary of an element does not meet the token limit of the processing window of the large language model, iteratively sort the subcommunities of the high-level community in descending order according to the number of tokens in the element summary, and use the subcommunity summary to replace the corresponding element summary that does not meet the token limit of the processing window of the large language model until it meets the token limit of the processing window of the large language model.

8. The large language model question answering method based on retrieval enhancement generation scheme according to claim 7 is characterized in that Step S6 generates a community answer based on the community-level summary obtained in step S5, and reduces it to obtain a global answer, which specifically includes the following steps: Get questions from users of the large language model; Based on the questions raised by users, several community-level summaries generate several community answers in parallel; The large language model scores the community answers obtained and deletes the community answers with scores lower than the set value; For the retained community answers, the relevance scores of the retained community answers are calculated, and the best answers are added to the processing window of the large language model until the token limit of the processing window of the large language model is reached to generate the final global answer.

9. A system for implementing the large language model question answering method based on the retrieval enhancement generation scheme as described in any one of claims 1 to 8, characterized in that It includes a document segmentation module, an instance extraction module, an instance conversion module, a graph community construction module, a summary generation module and a question-answering module; the document segmentation module, the instance extraction module, the instance conversion module, the graph community construction module, the summary generation module and the question-answering module are connected in series in sequence; the document segmentation module is used to segment the source document into a number of text blocks, and upload the data information to the instance extraction module; the instance extraction module is used to extract element instances from the obtained text blocks according to the received data information, and upload the data information to the instance conversion module; the instance conversion module is used to convert the obtained element instances into readable element summaries according to the received data information, and upload the data information to the graph community construction module; the graph community construction module is used to construct the obtained element summaries into graph communities according to the received data information based on the graph theory scheme, and upload the data information to the summary generation module; The summary generation module is used to generate a community-level summary for the obtained graph community based on the received data information, and upload the data information to the question-answering module; The question-answering module is used to generate community answers based on the received data information and the obtained community-level summary, and to reduce them to obtain global answers, thereby completing large language model question-answering based on the retrieval-enhanced generation scheme.

Images