Graph retrieval enhanced text generation method based on table atlas database in data
By constructing a knowledge graph in the data middle platform graph library and matching it using JIEBA word segmentation and TF-IDF+ minimum editing distance algorithm, combined with the thinking chain reasoning template, the problem of large language models generating illusions or factual errors is solved, achieving more accurate and credible text generation enhance the application capabilities of large language models in professional fields.
Patent Information
- Application Number
- CN202411238003.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-04
- Publication Date
- 2024-12-27
AI Technical Summary
Large language models may produce illusions or factual errors during text generation, which seriously restricts their application in professional fields.
Using graph retrieval enhancement text generation method based on the data middle platform map library, the knowledge graph is constructed through the graph construction module, the word segmentation matching and recommendation is used using JIEBA word segmentation and TF-IDF+ minimum editing distance algorithm, and the thinking chain reasoning template is used to generate accurate and trustworthy text content.
It improves the accuracy and credibility of text generation, reduces the occurrence of illusions or factual errors, provides more accurate information support, and generates expert-level corporate insights and understanding for large language models.
Smart Images
Figure CN119204112A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of graph processing technology, and specifically to a graph retrieval enhanced text generation method based on a data center graph library. Background Art
[0002] In the era of big models, activating the "emergence" capabilities in big models and adapting them to specific domain scenarios is the key to building competitiveness in vertical fields in the future, and high-quality domain data and knowledge are indispensable. Combining vertical domain data with knowledge graphs, designing appropriate prompt mechanisms, and fully stimulating the capabilities of big language models are conducive to achieving a "leap" in vertical domain task capabilities. However, big language models may also produce illusions or factual errors in the text generation process, which seriously restricts the application of big language models in professional fields.
[0003] Graph Retrieval Augmented Text Generation (GraphRAG) technology is a technology that combines retrieval technology and language generation technology to enhance the generation process. It can effectively utilize the structured characteristics of knowledge graphs, organize data in the form of nodes and relationships, increase the depth and associated context of retrieved information, thereby helping large language models generate more accurate, relevant and diverse information to meet user needs, and reduce the problems of illusions or factual errors that may be generated by large language models in applications.
[0004] As a unified platform that integrates data processing, storage, management and analysis, the data center provides support for various data applications based on the most complete data content of the enterprise. As a data organization form of the data center, the graph library has a rich knowledge system for specific tasks and specific fields. Combining the data center graph library data with the big model technology is conducive to giving full play to the advantages of the data center and the big model technology, so that it can demonstrate expert-level enterprise insight and understanding while exerting the general capabilities of the big model.
[0005] The present application designs a graph retrieval enhanced text generation technology based on the data middle platform graph library. It can use the data in the graph library as the professional knowledge base of the large language model. For "small problems" in the field, the graph retrieval enhanced text generation technology is used to retrieve the data in the data middle platform graph library, and the search results are combined with the "small problems" in the field and input into the large language model system. Through the chain of thinking and knowledge reasoning, more precise information is provided to the large language model, and more accurate and credible text content is generated, reducing the problem of illusions or factual errors that may be generated by the large language model. Summary of the invention
[0006] 1. Technical issues to be resolved
[0007] In view of the shortcomings of the prior art, the present invention provides a graph retrieval enhanced text generation method based on a data center graph library to improve the accuracy of generated content and reduce the problem of illusions or factual errors that may be generated by large language models.
[0008] (II) Technical solution
[0009] To achieve the above objectives, the present invention is implemented through the following technical solutions: a graph retrieval enhanced text generation method based on a data center graph library, specifically comprising the following steps:
[0010] S1. Users need to use the graph construction module to perform semi-automatic graph construction and index creation, build a unified knowledge graph perspective, and create an index for knowledge query during the graph library construction process, and store the index in the index library;
[0011] Index creation is divided into two parts: the index between keywords and entities, and the index between segmentation words and keywords. The index between segmentation words and keywords is used to map the relationship between "small problems" in the field and keywords. One segmentation word corresponds to multiple keywords, and one keyword also corresponds to multiple segmentations.
[0012] S2. Perform JIEBA word segmentation on the domain "small problem". Input the domain "small problem" into the JIEBA word segmentation tool, which will automatically perform word segmentation on the input text;
[0013] The word segmentation results will be output in the form of a word segmentation result list, where each word segmentation result contains the word segmentation result and position information;
[0014] S3, combining the segmentation and the entity, constructing the keyword set corresponding to the segmentation, constructing the entity set corresponding to the keyword, and constructing the entity set corresponding to the segmentation according to the segmentation-keyword set and the keyword-entity set;
[0015] S4, matching, recommending, filtering operations are performed from the word segmentation-entity set, and the final entity is selected;
[0016] S5. Combine graph entities and questions, query and reason about the questions. In this process, entities and questions are concatenated and returned to the large language model together with the thinking chain reasoning template to generate more accurate and credible text content.
[0017] Preferably, the index between the keywords and entities in step S1 is used to map the relationship between the knowledge graph entities and keywords, one keyword corresponds to multiple entities, and one entity also corresponds to multiple keywords.
[0018] Preferably, in step S2, the word segmentation result is further processed with the stop words and the synonym list, the word segmentation result is matched with the stop word list, the successfully matched stop word segmentations are deleted, the word segmentation result is matched with the synonym list, and the successfully matched synonymous word segmentations are replaced to obtain a more accurate word segmentation result.
[0019] Preferably, the specific steps of matching, recommending, and filtering operations from the word segmentation-entity set in step S4 are as follows:
[0020] T1. Matching: For the segmentation-entity set, the minimum edit distance algorithm is used to delete the unmatched segmentations, ensure that the order of appearance of the segmentation words is the same as the order of appearance of the words in the entity name, and calculate the distance between the segmentation and the entity name;
[0021] T2. Recommendation: Use the TF-IDF algorithm to recommend entities matched by the segmentation, and output them in order of similarity from high to low. We can set a threshold. When the similarity is less than the threshold, the entity corresponding to the segmentation is not recommended. For the recommended entities, the maximum number of recommended entities can be set;
[0022] T3. Filtering: For multiple entities with the same name recommended by word segmentation, manually filter out the wrong entity information by returning prompt information to the user.
[0023] Preferably, the minimum edit distance algorithm in step T1 refers to the minimum number of editing operations required to convert two strings from one to another. The smaller the edit distance between two strings, the more similar they are. If the two strings are equal, their edit distance is 0. Therefore, the minimum edit distance between the word segmentation and the entity name is calculated. The smaller the edit distance value, the more similar the word segmentation and the entity name.
[0024] The minimum edit distance algorithm is as follows:
[0025] distance(entity,tokens)=L1(entity.name)+L2(tokens)-2*L3
[0026] (entity.name,tokens);
[0027] Wherein, distance(entity,tokens) represents the minimum edit distance calculation function between the entity name and all tokens, entity represents the entity, entity.name represents the entity name, tokens represents all tokens, and L1(entity.name) is the function for calculating the length of the entity name;
[0028] L1(entity.name)=length(entity.name);
[0029] L2(tokens) is a function that calculates the total length of all tokens;
[0030] L2(tokens)=∑ token∈tokens length(token);
[0031] L3(entity.name,tokens) is the length function of the maximum common subsequence;
[0032]
[0033] Preferably, the TF-IDF algorithm in step T2 is specifically as follows: the similarity function between the entity and the word segmentation is as follows:
[0034]
[0035] Among them, similarity(entity,tokens) represents the similarity calculation algorithm between the entity name and all the tokens, distance(entity,tokens) represents the minimum edit distance calculation function between the entity name and all the tokens, length(entity.name) represents the entity name length calculation function, ∑ entity∈entities 1 token.word∈cut(entity.name) Indicates the frequency of token segmentation, ∑ entities 1 indicates the number of entities.
[0036] Preferably, the thought chain reasoning template in step S5 is based on the user's problem requirements, and presets a series of semantically coherent and logical intermediate reasoning steps, effectively establishing a bridging relationship between a group of user problem requirements and accurate text generation. Under the action of the thought chain prompt, the large language model generates the intermediate reasoning steps step by step, and finally generates more accurate, credible and reasoned text content.
[0037] (III) Beneficial effects
[0038] The present invention provides a graph retrieval enhanced text generation method based on a data center graph library. Compared with the prior art, it has the following beneficial effects: the graph retrieval enhanced text generation method based on a data center graph library adopts four main stages: pre-retrieval, retrieval, post-retrieval and generation: in the pre-retrieval stage, the user needs a graph construction module to pre-build a graph library, and in the process of building the graph library, an index for knowledge query is established; in the retrieval stage, for the "small problems" in the field, the "small problems" are segmented by JIEBA segmentation, and the segmentation results are matched and recommended by the TF-IDF+ minimum edit distance algorithm; in the post-retrieval stage, the matched and recommended entities are filtered to further improve the quality of text generation; in the generation stage, the filtered entities and problems are spliced and returned to the large language model together with the thinking chain reasoning template to generate more accurate and credible text content. The system can obtain data from the data center graph library, provide more accurate information for the large language model, generate more accurate and credible text content, and reduce the problem of illusion or factual error content that may be generated by the large language model. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a schematic diagram of the module description framework of the present invention;
[0040] Figure 2 A flow chart is designed for the graph retrieval enhanced text generation technology based on the data center graph library of the present invention. DETAILED DESCRIPTION
[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0042] See also Figure 1-2 , the embodiment of the present invention provides a technical solution: a graph retrieval enhanced text generation method based on a data center graph library, a graph retrieval enhanced text generation technology based on a data center graph library includes four main stages: pre-retrieval, retrieval, post-retrieval and generation, see Figure 1 :
[0043] In the pre-search phase, users need the graph construction module to pre-build a graph library and, during the graph library construction process, to establish an index for knowledge query;
[0044] In the retrieval stage, for the "small problems" in the field, the JIEBA word segmentation is used to segment the "small problems", and the TF-IDF+minimum edit distance algorithm is used to match and recommend knowledge for the segmentation results;
[0045] In the post-retrieval stage, the matched and recommended entities are filtered to further improve the quality of text generation;
[0046] In the generation phase, the filtered entities and questions are concatenated and returned to the large language model together with the thought chain reasoning template to generate more accurate and credible text content.
[0047] The design process of the graph retrieval enhanced text generation technology based on the data center graph library provided by the present invention includes:
[0048] 1. Users need to use the graph construction module to perform semi-automatic graph construction and index creation, build a unified knowledge graph perspective, and create an index for knowledge query during the graph library construction process, and store the index in the index library;
[0049] Index creation is divided into two parts: the index between keywords and entities, and the index between word segmentation and keywords.
[0050] The indexing function between keywords and entities is to map the relationship between knowledge graph entities and keywords. One keyword can correspond to multiple entities, and one entity can also correspond to multiple keywords.
[0051] The indexing function between participles and keywords is to map the relationship between "small problems" in the field and keywords. One participle can correspond to multiple keywords, and one keyword can also correspond to multiple participles.
[0052] 2. Perform JIEBA word segmentation on the domain "small problems". Input the domain "small problems" into the JIEBA word segmentation tool, which will automatically perform word segmentation on the input text;
[0053] The word segmentation results will be output in the form of a word segmentation result list, where each word segmentation result contains the word segmentation result and position information;
[0054] The word segmentation results are further processed with the stop words and synonym list. The word segmentation results are matched with the stop word list, and the successfully matched stop words are deleted. The word segmentation results are matched with the synonym list, and the successfully matched synonymous words are replaced to obtain more accurate word segmentation results.
[0055] 3. Combine word segmentation with entities to construct a keyword set corresponding to the word segmentation, and construct an entity set corresponding to the keyword. Construct the entity set corresponding to the word segmentation based on the word segmentation-keyword set and keyword-entity set.
[0056] 4. Perform matching, recommendation, and filtering operations from the word segmentation-entity set, and select the final entity;
[0057] Matching: For the word segmentation-entity set, the minimum edit distance algorithm is used to delete the unmatched word segmentations, ensure that the order of appearance of the word segmentation is the same as the order of appearance of the words in the entity name, and calculate the minimum edit distance between the word segmentation and the entity name;
[0058] The minimum edit distance algorithm refers to the minimum number of edit operations required to convert two strings from one to another. The smaller the edit distance between two strings, the more similar they are. If two strings are equal, their edit distance is 0. Therefore, the minimum edit distance between the word segmentation and the entity name is calculated. The smaller the edit distance value, the more similar the word segmentation and the entity name.
[0059] The minimum edit distance algorithm is as follows:
[0060] distance(entity,tokens)
[0061] =L1(entity.name)+L2(tokens)-2*L3(entity.name,tokens)
[0062] Among them, distance(entity,tokens) represents the minimum edit distance calculation function between the entity name and all tokens, entity represents the entity, entity.name represents the entity name, and tokens represents all tokens.
[0063] L1(entity.name) is a function that calculates the length of the entity name.
[0064] L1(entity.name)=length(entity.name)
[0065] L2(tokens) is a function that calculates the total length of all tokens.
[0066] L2(tokens)=Σ token∈tokens length(token)
[0067] L3(entity.name,tokens) is the length function of the maximum common subsequence.
[0068]
[0069] Recommendation: Use the TF-IDF algorithm to recommend entities matched by the word segmentation, and output them in order of similarity from high to low. We can set a threshold. When the similarity is less than the threshold, the entity corresponding to the word segmentation is not recommended. For the recommended entities, the maximum number of recommended entities can be set.
[0070] The similarity function between entities and word segments is as follows:
[0071]
[0072] Among them, similarity(entity,tokens) represents the similarity calculation algorithm between the entity name and all the tokens, distance(entity,tokens) represents the minimum edit distance calculation function between the entity name and all the tokens, length(entity.name) represents the entity name length calculation function, ∑ entity∈entities 1 token.word∈cut(entity.name) Indicates the frequency of token segmentation, ∑ entities 1 indicates the number of entities.
[0073] Filtering: For multiple entities with the same name recommended by word segmentation, incorrect entity information is manually filtered out by returning prompt information to the user.
[0074] 5. Combine graph entities and questions, query and reason about the questions. In this process, the entities and questions are concatenated and returned to the large language model together with the thinking chain reasoning template, ultimately generating more accurate, credible and reasoned text content.
[0075] The thought chain reasoning template is based on the user's problem needs, and presets a series of semantically coherent and logical intermediate reasoning steps, effectively establishing a bridge relationship between a group of user problem needs and accurate text generation. Under the action of the thought chain prompt, the large language model can generate intermediate reasoning steps step by step, and finally generate more accurate, credible and reasoned text content.
[0076] Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.
[0077] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0078] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A graph retrieval enhanced text generation method based on a data center graph library, characterized by: The specific steps include: S1. Users need to use the graph construction module to perform semi-automatic graph construction and index creation, build a unified knowledge graph perspective, and create an index for knowledge query during the graph library construction process, and store the index in the index library; Index creation is divided into two parts: the index between keywords and entities, and the index between segmentation words and keywords. The index between segmentation words and keywords is used to map the relationship between "small problems" in the field and keywords. One segmentation word corresponds to multiple keywords, and one keyword also corresponds to multiple segmentations. S2. Perform JIEBA word segmentation on the field "small problem". Input the field "small problem" into the JIEBA word segmentation tool, which will automatically perform word segmentation on the input text; The word segmentation results will be output in the form of a word segmentation result list, where each word segmentation result contains the word segmentation result and position information; S3, combining the segmentation and the entity, constructing the keyword set corresponding to the segmentation, constructing the entity set corresponding to the keyword, and constructing the entity set corresponding to the segmentation according to the segmentation-keyword set and the keyword-entity set; S4, matching, recommending, filtering operations are performed from the word segmentation-entity set, and the final entity is selected; S5. Combine graph entities and questions, query and reason about the questions. In this process, entities and questions are concatenated and returned to the large language model together with the thinking chain reasoning template to generate more accurate and credible text content.
2. According to claim 1, a graph retrieval enhanced text generation method based on a data center graph library is characterized by: The index between keywords and entities in step S1 is used to map the relationship between knowledge graph entities and keywords. One keyword corresponds to multiple entities, and one entity also corresponds to multiple keywords.
3. According to claim 1, a graph retrieval enhanced text generation method based on a data center graph library is characterized by: In step S2, the word segmentation result is further processed with the stop words and the synonym list, the word segmentation result is matched with the stop word list, the successfully matched stop word segmentations are deleted, the word segmentation result is matched with the synonym list, and the successfully matched synonymous word segmentations are replaced to obtain a more accurate word segmentation result.
4. According to claim 1, a graph retrieval enhanced text generation method based on a data center graph library is characterized by: The specific steps of matching, recommending, and filtering operations from the word segmentation-entity set in step S4 are as follows: T1. Matching: For the segmentation-entity set, the minimum edit distance algorithm is used to delete the unmatched segmentations, ensure that the order of appearance of the segmentation words is the same as the order of appearance of the words in the entity name, and calculate the distance between the segmentation and the entity name; T2. Recommendation: Use the TF-IDF algorithm to recommend entities matched by the segmentation, and output them in order of similarity from high to low. We can set a threshold. When the similarity is less than the threshold, the entity corresponding to the segmentation is not recommended. For the recommended entities, the maximum number of recommended entities can be set; T3. Filtering: For multiple entities with the same name recommended by word segmentation, incorrect entity information is manually filtered out by returning prompt information to the user.
5. According to claim 4, a graph retrieval enhanced text generation method based on a data center graph library is characterized in that: The minimum edit distance algorithm in step T1 refers to the minimum number of editing operations required to convert two strings from one to another. The smaller the edit distance between two strings, the more similar they are. If the two strings are equal, their edit distance is 0. Therefore, the minimum edit distance between the word segmentation and the entity name is calculated. The smaller the edit distance value, the more similar the word segmentation and the entity name. The minimum edit distance algorithm is as follows: distance(entity,tokens)=L1(entity.name)+L2(tokens)-2*L3 (entity.name,tokens); Wherein, distance(entity,tokens) represents the minimum edit distance calculation function between the entity name and all tokens, entity represents the entity, entity.name represents the entity name, tokens represents all tokens, and L1(entity.name) is the function for calculating the length of the entity name; L1(entity.name)=length(entity.name); L2(tokens) is a function that calculates the total length of all tokens; L2(tokens)=∑ token∈tokens length(token); L3(entity.name,tokens) is the length function of the maximum common subsequence; 6. According to claim 4, a graph retrieval enhanced text generation method based on a data center graph library is characterized in that: The TF-IDF algorithm in step T2 is specifically as follows: the similarity function between entities and word segments is as follows: Among them, similarity(entity,tokens) represents the similarity calculation algorithm between the entity name and all the tokens, distance(entity,tokens) represents the minimum edit distance calculation function between the entity name and all the tokens, length(entity.name) represents the entity name length calculation function, ∑ entity∈entities 1 token.word∈cut(entity.name) Indicates the frequency of token segmentation, ∑ entities 1 indicates the number of entities.
7. According to claim 1, a graph retrieval enhanced text generation method based on a data center graph library is characterized by: The thought chain reasoning template in step S5 is based on the user's problem requirements, and presets a series of semantically coherent and logical intermediate reasoning steps, effectively establishing a bridge relationship between a group of user problem requirements and accurate text generation. Under the action of the thought chain prompt, the large language model generates the intermediate reasoning steps step by step, and finally generates more accurate, credible and reasoned text content.