Code language model-based generative knowledge graph construction method and device
By designing structure-aware code suggestion templates and reconstructing pre-trained code language models, the problem of ignoring structural information in existing methods is solved, achieving more efficient relation triple extraction and knowledge graph construction.
Patent Information
- Application Number
- CN202310922506.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-07-26
- Publication Date
- 2026-01-16
- Estimated Expiration
- 2043-07-26
AI Technical Summary
Existing pipeline and end-to-end methods tend to ignore structural information when extracting relation triples from unstructured text, resulting in poor extraction performance. Code language models outperform natural language models in handling structured tasks, but existing methods do not fully utilize their structural understanding capabilities.
We design structure-aware code suggestion templates, convert knowledge graphs into code form, generate relation triples using code language models, explicitly model structural information, label text information using code suggestion templates and text suggestion templates, and improve the model's structural understanding ability through pre-training and fine-tuning of small and large code language models.
By effectively utilizing the structural understanding capabilities of code language models, the accuracy and structural understanding of relation triple extraction are improved, thereby enhancing the effectiveness of knowledge graph construction.
Smart Images
Figure CN117033653B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application belongs to the technical field of knowledge graph and machine learning, and particularly relates to a generative knowledge graph construction method and device based on a code language model. BACKGROUND
[0002] In the financial field, it is necessary to identify listed companies, products, and supplier-customer relationships from unstructured text corpus to construct a listed company industry chain knowledge graph. In the social field, it is necessary to identify the relationships between characters and institutions from unstructured text corpus to construct a social network graph to mine the potential relationships between characters and institutions. In the risk control field, it is necessary to identify the relationships between characters, debts, and events from unstructured text corpus to construct a risk control entity graph to mine the potential risk matters of the subject.
[0003] In the above three technical fields, extracting relationship triplets from unstructured text corpus, i.e., extracting entity pairs and their relationships, is crucial for the construction of large-scale knowledge graphs in each field. Existing pipeline methods gradually combine named entity recognition, entity linking, and relationship extraction, such as CasRel and PURE, and have achieved good results. However, the pipeline method inevitably accumulates errors. As a result, another end-to-end method has emerged. The end-to-end method converts the relationship triplet structure into a serialized text, and then uses a neural network with expressive power, such as a pre-trained language model (LM), to directly input the unstructured text and output the predicted encoded relationship triplet in flat text. Although these previous methods have achieved good results, the existing methods usually "serialize" the output triplet structure into flat text, inevitably ignoring some structural information. Language models have difficulty generating outputs that do not conform to natural text grammar and structure, because this type of text is very different from the form of the pre-training corpus.
[0004] Despite these obstacles, recent progress in code large language models (Code-LLMs) has demonstrated the potential of utilizing Code-LLMs for complex reasoning, structured common sense reasoning, and structure prediction tasks. Code language models (such as UniXcoder, CodeBERT, Codex, etc.) are pre-trained on mixed corpora of natural text and code, exhibiting impressive code generation capabilities. For problems requiring deeper reasoning, the emergence of code language models provides new solutions. Some recent works (such as CoCoGen, Code4Struct) convert natural language in the task into code form, thereby reconstructing the task into a code generation task, and achieve great success by utilizing code language models. These works show that code language models have higher structural understanding capabilities after converting task data into appropriate code structures, and are superior to natural language models in handling structured tasks. In addition, the reStructured Pre-training paradigm emphasizes the role of data, and the pre-training of the model and the fine-tuning of the downstream task are regarded as data storage and access processes. From the perspective of this paradigm, it is assumed that pre-training code language models on the same code form data as the downstream task may further improve the structural understanding capabilities of the model for specific tasks. SUMMARY
[0005] In view of the above, the purpose of the present application is to provide a code language model-based generative knowledge graph construction method and device to perform relation triple extraction tasks in a code generation manner, enhance the pre-trained code language model by explicitly modeling structural information, improve the structural understanding capabilities of the code language model, and further improve the performance of relation triple extraction.
[0006] To achieve the above-mentioned purpose of the application, the code language model-based generative knowledge graph construction method provided by the embodiment comprises the following steps:
[0007] A structure-aware code prompt template is designed, which includes a schema prompt template and a text prompt template. The schema prompt template is used to convert a knowledge graph into a code form, including a base class Entity, a base class Relation, a custom subclass inheriting the base class Entity and / or the base class Relation, a Triple class, and an Extract class. The text prompt template is used to convert text into a code form, specifically using special flags to mark text information.
[0008] Given a sample data pair containing input text and known knowledge graph, the sample data pair is converted into a code prompt using the code prompt template, and the test data containing only test text is also converted into a code prompt.
[0009] The sample data pair and the test data of the code prompt are input into a large code language model, code triplets are generated based on the prompt learning mode, the code is converted into structural triplets, and a knowledge graph is constructed based on the structural triplets.
[0010] The knowledge graph is a financial domain knowledge graph, a social domain knowledge graph, and a risk control domain knowledge graph, that is, the construction of the financial domain knowledge graph, the social domain knowledge graph, and the risk control domain knowledge graph is realized.
[0011] Preferably, in the schema prompt template, the base class Entity and the base class Relation are used for general definition of entities and relations in the knowledge graph; the Triple class is composed of (head entity, relation, tail entity) and is used for instance conversion of triplets; the Extract class is used for conversion of multiple triplets, and the Extract class is instantiated with a list of Triple class instances as input.
[0012] Preferably, when the sample data pair is converted into code form by using the code prompt template, the input text is marked based on the special flag of the text prompt template to form the context prompt in code form, and the known knowledge graph is filled into the schema prompt template as an instance to form the schema prompt in code form.
[0013] When the test data containing only test text is converted into code form by using the code prompt template, the test text is marked based on the special flag of the text prompt template to form the task text prompt in code form.
[0014] Preferably, the method further comprises training a small code language model and applying it to construct a knowledge graph in comparison with a large code language model, specifically obtaining a sample data pair from a relation extraction corpus or obtaining a sample data from a downstream actual extraction task, and converting the sample data pair into a code prompt by using a code prompt template, inputting the sample data pair in code prompt form into the small code language model for reconfiguration pre-training and supervision training based on real labels to optimize the parameters of the small code language model, generating code of encoded triplets based on test data of the code prompt by using the parameter-optimized small code language model, converting the code into structural triplets, and constructing a knowledge graph based on the structural triplets.
[0015] The extraction task is a triplet extraction task in the financial domain, the risk control domain, and the social domain.
[0016] Preferably, the method further comprises adding an intermediate reasoning step, inputting the intermediate reasoning step-by-step extraction prompt into the large code language model by adding the step-by-step extraction prompt of the relationship and entity in the triple in the code prompt, or training the small code language model using the data with the added intermediate reasoning step, and the model performs step-by-step extraction of entities and relationships based on the step-by-step extraction prompt during reasoning, and then forms code of the coded triple based on the extraction result.
[0017] Preferably, the large code language model adopts a CODEX model.
[0018] Preferably, the small code language model adopts a CodeT5 model.
[0019] Preferably, it further comprises calculating a performance index of the model, and evaluating the extraction performance of the large code language model or the small code language model based on the performance index, wherein the performance index is the gap between the predicted triple extracted by the model and the real triple.
[0020] To achieve the above-mentioned purposes of the application, the embodiment also provides a code language model-based generative knowledge graph construction device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the code language model-based generative knowledge graph construction method when executing the computer program.
[0021] Compared with the prior art, the application has at least the following technical effects:
[0022] The code prompt template is constructed, and the effect of this code structure prompt is better than that of pure text prompt. The structural understanding ability of the code language model is effectively utilized, and the language model pre-trained only on natural format text is no longer used. The relationship triple extraction task is solved by using the code form structure prompt and the code language model. The structural understanding ability of the model for the relationship triple extraction task is further improved by using the reconstructed code pre-training. BRIEF DESCRIPTION OF DRAWINGS
[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0024] Figure 1 is a flowchart of a code language model-based generative knowledge graph construction method provided by the embodiment;
[0025] Figure 2The code prompt structure schematic diagram is provided by the embodiment. DETAILED DESCRIPTION
[0026] To make the objectives, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the protection scope of the present application.
[0027] In view of the need to identify listed companies, products, supplier and customer relationships in the financial field, construct a listed company industry chain knowledge graph, penetrate knowledge for policy and information, and mine the intrinsic value; it is also found in the social field to identify the relationship between persons and institutions, construct a social network graph, and mine the potential relationship between persons and institutions; and in view of the need to identify the relationship between persons, debts and events in the risk control field, construct a risk control entity graph, and mine the potential risk matter relationship of the subject. The embodiment of the present application provides a code language model-based generative knowledge graph construction method and device, which realizes the construction of various knowledge graphs.
[0028] As shown in Figure 1 , the code language model-based generative knowledge graph construction method provided by the embodiment includes the following steps:
[0029] Step 1, design a code prompt template with structure perception.
[0030] In the embodiment, the knowledge graph construction task is restructured as a code generation problem. For each sample data pair in the knowledge graph construction task, it is represented in the form of , where is the input text, is the knowledge graph to be generated, and more specifically, is a triple containing entities and their relationships. Since the language model trained on natural language text may not be able to capture the topological structure of the relationship, the core idea of the present application is to convert the output triple (graph ) into a semantically equivalent programming language (graph ) written in Python. The programming language has better representation of structural information, so that the code language model with stronger structural understanding ability can be used to do this task in the form of code generation. The present application uses a pre-defined code prompt template conforming to Python syntax to convert the original data form into the code form The restructured code form will be fed into the code language model for reasoning to obtain the code form output Then it is converted into the original form of .
[0031] In an embodiment, the code prompt template includes a schema prompt template and a text prompt template, wherein the schema prompt template is used to convert the knowledge graph into code form, including a base class Entity, a base class Relation, a custom subclass inheriting the base class Entity and / or the base class Relation, a Triple class, and an Extract class, and the text prompt template is used to convert the text into code form, specifically using special flags to mark the text information. Through the code prompt template, the sample data pair can be converted into a series of predefined classes in Python, and a text prompt containing a document string describing the task is constructed. Since the reconstructed code prompt retains the syntax and structural features, the code language model tends to generate more accurate relationships and entities.
[0032] The schema in the knowledge graph describes the structure of the data, including entities, relationships, attributes, and constraints that make up the knowledge graph. In an embodiment, the schema information is integrated by inheriting the Python class definition, as shown in Figure 2 As shown, the embodiment defines a base class Entity and a base class Relation, which describe the general definition of entities and relationships in knowledge graph construction. For example, to represent an entity belonging to the "Person" type, the Person class should inherit the base class Entity, and therefore, a custom subclass inheriting the base class is also constructed. The embodiment further defines a Triple class, which is used to represent triples contained in the text input. The Triple class is composed of a head-tail entity pair and their corresponding relationship. Each triple will be represented as an instance of its corresponding entity and relationship type class. For example, if there is a triple (London, located in, UK), it is represented in the form of code designed by the invention—Triple(LOC(“London”), Rel(“located in”), LOC(“UK”)), and the Extract class instance needs to input a list of Triple instances as parameters when instantiated to extract multiple triples in the text.
[0033] Step 2, given a sample data pair containing input text and known knowledge graph, the sample data pair is converted into a code prompt using the code prompt template, and the test data containing only the test text is converted into a code prompt.
[0034] In an embodiment, to perform a task on a never-seen sample data pair, a large code language model is provided with several sample data pairs of complete basic fact relationship triples composed of task prompts. k samples in the dataset are selected, which contain all n relationship types, and they are spliced with the task prompt for few-shot learning.
[0035] When performing few-shot learning, the sample data pair is converted into code hints using the code hint template. The specific process is as follows: as shown in Figure 2 the special flag bit of the text hint template marks the input text to form a context hint in code form, and the known knowledge graph is filled into the schema hint template to form a schema hint in code form.
[0036] At the same time of few-shot learning, test generation is also performed. Specifically, the test data containing only test text is converted into code form using the code hint template, that is, the special flag bit of the text hint template marks the test text to form a task hint in code form.
[0037] Step 3: input the code hint sample data pair and test data into the large code language model and apply it to build the knowledge graph.
[0038] In the embodiment, the code hint sample data pair converted by the code hint template in step 2 is input into the large code language model, and the code containing the relationship triple information is generated based on the prompt learning method, and the code is converted into a structure triple, and the knowledge graph is constructed based on the structure triple. Preferably, the large code language model adopts a 100B parameter level CODEX model.
[0039] The process of prompting the CODEX model to generate the output required for the task, i.e., the code containing the relationship triple information, is as follows: first, give the model an instruction, such as "complete the following given code" or "continue to generate the second half of the code based on the first half of the given code", then input the task hint in Python form containing the input text information and the Python module capable of representing the relationship triple structure information (as well as the context learning sample) into the code large language model, and prompt the CODEX model to generate the code containing the relationship triple information.
[0040] Step 4: train the small code language model and apply it to build the knowledge graph.
[0041] When using the small code language model to perform relationship triple extraction tasks, since the parameter quantity and structure understanding ability of the small language model are much lower than those of the large code language model, it is necessary to fine-tune the small language model. In the embodiment, the CodeT5 model is preferably used as the small code language model to increase the understanding ability of the small code language model for the relationship structure.
[0042] Specifically, the fine-tuning and constructing knowledge graph process is: obtaining sample data pairs of actual extraction tasks from downstream, converting the sample data pairs into code prompts by using a code prompt template, inputting the sample data pairs of code prompts into a small code language model for supervised training based on real labels to optimize parameters of the small code language model, encoding code of a test data triple based on the code prompt by using the parameter-optimized small code language model, converting the code into a structured triple, and constructing a knowledge graph based on the structured triple.
[0043] In the embodiment, although the small code language model is fine-tuned on some general code, such as CodeT5 obtained by pre-training T5 on some general code, the small code language model still has great difficulty in understanding the complex code structure of the relation triple extraction, and the gap with the large code language model is still large. In order to improve the performance of the small code language model on the relation triple extraction task, the present application refers to the reStructured Pre-training paradigm, which emphasizes the role of data, and the pre-training of the model and the fine-tuning of the downstream task are regarded as the process of data storage and access. From the perspective of this paradigm, it is assumed that pre-training the code language model on the same code form data as the downstream task may further improve the model's structural understanding ability for specific tasks such as relation triple extraction. Specifically, large-scale relation extraction corpus, including T-Rex, TekGEN, KELM dataset, is reconstructed into a code prompt template format, and the small code language model (including T5 and CodeT5) is pre-trained on these corpora to further improve the model's structural understanding ability for the relation triple extraction task. After the restructured code pre-training is completed, fine-tuning and testing on the downstream task are continued.
[0044] Step 5, calculate the performance index of the model.
[0045] In the embodiment, the code of the encoded relation triple generated by the large code language model or the small code language model is decoded into the form of normal text, i.e. the predicted triple, and the gap between the predicted triple and the real label is calculated as the performance index. The specific performance index is relation strict micro F1. Based on this performance index, the extraction performance can be evaluated.
[0046] In this embodiment, inspired by the concept of thought chains, optional intermediate inference steps can be added to steps 1-5 above to improve the inference capabilities of large or small code language models. Specifically, by adding step-by-step extraction hints for relations and entities to the text and extracted triples in the code suggestions, the step-by-step extraction hints for intermediate inference are input into the large or small code language model. During the inference process, the model extracts entities and relations step by step based on the step-by-step extraction hints, and then forms triple code based on the extraction results.
[0047] Specifically, given text input in code form Code hints for sample data pairs and probability model p LM The purpose of the embodiments is to maximize Likelihood:
[0048]
[0049] Among them, g i yes The i-th token, express The length to be generated. For few-sample hints... From k samples For composition. For example Figure 2 As shown, the embodiment incorporates an explicit intermediate step to enhance the reasoning ability of the language model. This reasoning step... Add to code completion. therefore:
[0050]
[0051] in, and Defined as:
[0052]
[0053]
[0054] Where, r i It is all One step in a reasoning process. Specifically, complex knowledge graph reasoning tasks are broken down into multiple steps, such as... Figure 2 As shown, the entities contained in the text are generated first, then the possible relations in the text are generated, and finally the triples are generated.
[0055] The code language model-based generative knowledge graph construction method provided by the above embodiments can effectively capture the structural information of the knowledge graph, and thus improve the accuracy of the triple extraction, compared with the traditional method of using natural text.
[0056] Based on the same inventive concept, the embodiments also provide a code language model-based generative knowledge graph construction device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the code language model-based generative knowledge graph construction method is implemented, which comprises the following steps:
[0057] Step 1: design a code prompt template with structural perception;
[0058] Step 2: given a sample data pair comprising an input text and a known knowledge graph, convert the sample data pair into a code prompt using the code prompt template, and convert the test data comprising only the test text into a code prompt;
[0059] Step 3: input the code prompt sample data pair and the test data into a large code language model and apply it to construct a knowledge graph;
[0060] Step 4: train a small code language model and apply it to construct a knowledge graph;
[0061] Step 5: calculate the performance indicators of the model.
[0062] In practical applications, the memory can be a proximal volatile memory such as RAM, or a non-volatile memory such as ROM, FLASH, floppy disk, mechanical hard disk, etc., or a remote storage cloud. The processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), i.e., the steps of the code language model-based generative knowledge graph construction method can be implemented through these processors.
[0063] The above specific embodiments have described the technical solutions and advantages of the present application in detail. It should be understood that the above description is only the most preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, supplements, and equivalent replacements made within the principle range of the present application shall be included in the protection scope of the present application.
Claims
1. A code language model based generative knowledge graph construction method, characterized in that, The method comprises the following steps: a code prompt template for structure perception is designed, the code prompt template comprises a schema prompt template and a text prompt template, the schema prompt template is used to convert a knowledge graph into a code form, and the text prompt template is used to convert text into a code form, and the text prompt template specifically adopts a special flag to mark text information; a sample data pair comprising input text and a known knowledge graph is given, the sample data pair is converted into a code prompt by using the code prompt template, and test data comprising only test text is converted into a code prompt; the sample data pair and the test data in the code prompt are input into a large code language model, code of an encoded triple is generated based on a prompt learning mode, the code is converted into a structural triple, and a knowledge graph is constructed based on the structural triple; a small code language model is trained and applied to construct a knowledge graph, which is compared with the large code language model, sample data pairs are obtained from a relation extraction corpus or actual extraction tasks are obtained from a downstream, the sample data pairs are converted into code prompts by using the code prompt template, the sample data pairs in the code prompt are input into the small code language model for code reconstruction pre-training and supervision training based on real labels, parameters of the small code language model are optimized, code of an encoded triple is generated based on test data in the code prompt by using the small code language model with optimized parameters, the code is converted into a structural triple, and a knowledge graph is constructed based on the structural triple; the extraction task is a triple extraction task in a financial field, a risk control field and a social field; performance indicators of the model are calculated, and extraction performance of the large code language model or the small code language model is evaluated based on the performance indicators, wherein the performance indicators are a gap between predicted triples extracted by the model and real triples; The knowledge graph is a knowledge graph in a financial field, a knowledge graph in a social field and a knowledge graph in a risk control field, that is, construction of the knowledge graph in the financial field, the knowledge graph in the social field and the knowledge graph in the risk control field is realized. 2.The code language model based generative knowledge graph construction method of claim 1, wherein, In the schema prompt template, the base class Entity and the base class Relation are used to define entities and relations in the knowledge graph; the Triple class is composed of (a head entity, a relation and a tail entity) and is used to convert triples; the Extract class is used to convert multiple triples, and the Extract class is instantiated by inputting a list of Triple class instances. 3.The code language model based generative knowledge graph construction method of claim 1, wherein, When the sample data pair is converted into a code form by using the code prompt template, the input text is marked based on the special flag of the text prompt template to form a context prompt in the code form, and the known knowledge graph is filled into the schema prompt template as an instance to form a schema prompt in the code form. When the test data containing only test texts is converted into code form by using a code prompt template, the test texts are marked based on a special flag of the text prompt template to form task text prompts in code form. 4.The code language model based generative knowledge graph construction method of claim 1, wherein, The method further comprises adding an intermediate reasoning step, inputting the step-by-step extraction prompts of the intermediate reasoning into a large code language model by adding the step-by-step extraction prompts of relations and entities in the triples in the code prompts, or training a small code language model using data with the added intermediate reasoning step, and the model performs step-by-step extraction of entities and relations based on the step-by-step extraction prompts during the reasoning process, and then forms code of the extracted triples based on the extraction results. 5.The code language model based generative knowledge graph construction method of claim 1, wherein, The large code language model adopts a CODEX model. 6.The code language model based generative knowledge graph construction method of claim 1, wherein, The small code language model adopts a CodeT5 model.
7. A code language model based generative knowledge graph construction apparatus, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor implements the code language model-based generative knowledge graph construction method of any one of claims 1-6 when executing the computer program.