An automatic construction system of a knowledge graph and a working method thereof

Abstract

The application discloses an automatic construction system and working method of a knowledge graph, comprising an information collection module, a schema generation module, an information extraction module, an information fusion module and a pre-training language model; the schema generation module and the information extraction module are used for expressing natural language texts in the form of a prompt deep learning model; the information collection module is used for collecting unstructured text corpus and corresponding schema information, including but not limited to unannotated internet text corpus; the pre-training language model is arranged in a server, and is trained in the form of a transformer-based encoder-decoder deep learning architecture, so as to perform self-supervised training on the text corpus; then, the pre-training language model is fed back to the schema generation module and the information extraction module after corresponding fine-tuning, so as to improve training accuracy; the application solves the problem that information extraction based on schema guidance depends on artificial given schema structure but cannot solve the problem that background information relied on by automatic construction of the schema and extraction is limited to current texts and cannot effectively utilize the constructed knowledge graph.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence knowledge graph technology, specifically relating to an automatic knowledge graph construction system and its working method. Background Technology

[0002] A knowledge graph is a graph structure that represents data using entities (nodes) – relations (edges) – entities (nodes). Both nodes and relations can be defined with multiple attributes to express complex data structures; for example, events can be expressed using time, location, subject, and behavior on a relation.

[0003] Due to its unique graph structure, knowledge graphs offer significant efficiency improvements in data processing, searching, statistical analysis, and modeling involving relationships, leading to their widespread application. Traditional knowledge graph construction typically involves two main steps: defining the data structure (schema) of entities and relationships based on business requirements; and manually labeling or aligning unstructured or semi-structured data according to the specified schema, constructing an information extraction module, and extracting data from the actual data. Both steps heavily rely on manual methods, resulting in high construction costs and significant challenges in domain expansion and automatic updates.

[0004] The main methods for constructing knowledge graphs include: In defining the schema, there are manual definitions and methods utilizing the knowledge organization structures and preset patterns of encyclopedia pages, such as is-a type concept graphs; in the information extraction step, the main methods include lexical pattern-based methods, clustering-based methods, and schema-guided information extraction methods. Manual definition and lexical pattern-based methods rely on manually specified patterns, which are difficult to handle the complex and varied text expressions in reality. Utilizing encyclopedia knowledge organization can effectively solve the cold start problem, but its structured knowledge coverage is very small, still requiring other methods to mine information from unstructured text. Schema-guided information extraction is a deep learning model method, borrowing solutions from question-answering tasks: the schema expression is treated as the question, the text content to be extracted is treated as background information, and the answer to the question is the extracted structured data.

[0005] However, at present, schema-guided information extraction relies on manually provided schemas, which does not solve the problem that the background information on which the schema is automatically constructed and extracted is limited to the current text, and cannot effectively utilize the constructed knowledge graph. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an automatic knowledge graph construction system and its working method.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] An automatic knowledge graph construction system includes an information collection module, a schema generation module, an information extraction module, an information fusion module, and a pre-trained language model; the schema generation module and the information extraction module use a deep learning model that provides prompts to represent natural language text; the information collection module is used to collect unstructured text corpora and their corresponding schema information, including but not limited to unlabeled Internet text corpora.

[0009] The schema generation module consists of the same encoder and generative decoder as the pre-trained language model. The schema generation module is used to generate schema fragments represented as sequences, corresponding to the specified part of SELECT.

[0010] The information extraction module is used to extract natural language corpus and output structured information. Unstructured text and its corresponding schema information are concatenated as input, and the specific data values ​​corresponding to the text and schema are output. The structured information is expressed in SQL.

[0011] The pre-trained language model is set up on the server and trained using a deep learning architecture based on transformer encoder-decoder. It performs self-supervised training on text corpora and then feeds back to the schema generation module and information extraction module after appropriate fine-tuning to improve training accuracy.

[0012] The information fusion module is used to extract structured data containing partial schema information from a single unstructured text, and the credibility of the information source is low. When fusing fragments of structured data into the knowledge graph, the credibility of the information is verified.

[0013] Furthermore, the loss function between the encoder and decoder in the schema generation module is expressed as follows:

[0014]

[0015] Where, θ e θ d These are the parameters for the encoder and decoder, respectively; x is the original text; metaSQL is the meta information expressed in SQL language; schema is the schema information to be output.

[0016] Furthermore, the loss function of the information extraction module is:

[0017]

[0018] Where, θ e θ d These are the parameters for the encoder and decoder, respectively; x is the original text; schemaSQL is the schema information expressed in SQL language; and data is the structured data information to be output.

[0019] Furthermore, the loss function of the pre-trained language model is:

[0020] L text =∑ x -log p(x target |x;θ e θ d )

[0021] Where, θ e θ d These are the parameters for the encoder and decoder, respectively; x is the input text, x target For the text to be predicted, if a masked language model is used, x target That is, the obscured text; if a causal language model is used, x target That is, the next word, the next paragraph, or the next sentence.

[0022] Furthermore,

[0023] The information fusion module uses the following formula to verify the value of a certain attribute field key in the schema:

[0024] value = max k (∑ j weight j (value k (for k = 1, ..., N)

[0025] Where, weight j The credibility of source j is represented by the PageRank algorithm, calculated using this algorithm; k = 1, ..., N represents the N possible values ​​for value.

[0026] value k Represents the k-th value; the same value extracted from different sources j. k Each value has a different weight. After summing the weight values, the value with the highest corresponding score is taken. k That is, the value corresponding to the key.

[0027] The present invention also provides a working method for an automatic knowledge graph construction system, comprising the following steps: S1: pre-training a language model through a self-supervised method of enumeration of pre-trained language models, wherein the unstructured text corpus includes various unlabeled Internet text corpora;

[0028] S2: Collect unstructured text and its corresponding schema information as training data for the schema generation module; search for text containing structured values ​​through structured knowledge, and use this text and the schema of structured knowledge as training data.

[0029] S3: The unstructured text and its corresponding schema information are concatenated as input, and the specific data values ​​corresponding to the text and schema are output as the training of the information extraction module; and the specific output data format is given.

[0030] S4: After obtaining the schema generation module through S2 and the information extraction module through S3, a complete information extraction can be performed on a new document: First, obtain schema information through the schema generation module, and then obtain structured data information based on the text and schema information;

[0031] S5: After obtaining structured data information from any source through S4, the structured data is verified and fused using the knowledge fusion methods listed in the information fusion module; the verified original text and structured data are added to the training data, and continuous training is carried out through the training methods of the schema generation module and the information extraction module.

[0032] Preferably, the training data used by the information extraction module needs to simultaneously save the original text, the structured schema, and the structured data after the method in step S2.

[0033] Compared with the prior art, the present invention has the following beneficial effects:

[0034] Compared to the existing prompt design methods, using SQL expressions allows for the convenient addition of background information or constraints beyond the original text through WHERE expressions, while maintaining consistency with SQL syntax and facilitating continuous expansion.

[0035] Structured data integrated into knowledge graphs is not 100% reliable. As new information is added, older knowledge may no longer maintain its highest ranking and will be updated. The training data will also be updated accordingly, continuously improving the reliability of the extraction model. Attached Figure Description

[0036] Figure 1 This is a system architecture diagram of an automatic knowledge graph construction system according to the present invention;

[0037] Figure 2 This is a diagram illustrating the processing steps of the pre-trained language model of this invention;

[0038] Figure 3 This is a schematic diagram of the schema structuring process of the present invention;

[0039] Figure 4 This is a schematic diagram of the input text splicing process and result output of the present invention;

[0040] Figure 5 This is a schematic diagram of the results extracted by the information extraction module of the present invention;

[0041] Figure 6 This is a schematic diagram of the verification process of the information fusion module of the present invention;

[0042] Figure 7 This is an example of a verification result provided in an embodiment of the present invention;

[0043] Figure 8 This is Example 2 of the verification result provided in the embodiments of the present invention;

[0044] Figure 9 This is Example 3 of the verification results provided in this embodiment of the invention;

[0045] Figure 10 This is Example 4 of the verification results provided in this embodiment of the invention. Detailed Implementation

[0046] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0047] Example

[0048] In order to achieve the purpose of this invention, such as Figure 1 As shown, this embodiment provides an automatic knowledge graph construction system:

[0049] Expression rule definition in the information collection module

[0050] meta-schema representation:

[0051] A fixed meta structure of the schema is used on top of the variable schema.

[0052] The schema has a fixed meta structure, where the entity schema's meta structure is {type, value}, where type represents the entity type and value represents the entity name;

[0053] Relationships are divided into is-a and has-a, which express conceptual relations and possession relations, respectively.

[0054] Its meta structure consists of {type, value}, which express the relation name and the relation value;

[0055] The meta structure of an event schema is {who, what, where, when, why, how}, which respectively represent the event subject, event description, event location, event occurrence, event background, and event outcome.

[0056] Its meta structure is {type, value}.

[0057] The attributes of entities, relationships, and events change depending on the actual scenario.

[0058] For example, companies and schools have different entity attributes, and their related relationships, events, and other attributes are also different.

[0059] The prompt definition in the information collection module is as follows:

[0060] Both the schema generation module and the information extraction module are built based on a deep learning model that provides prompts. These models are used to represent natural language text, and their syntax follows the format SELECT#FROM#WHERE#.

[0061] Here, # represents the text content to be filled; SELECT represents the information to be extracted; and FROM can take four values: entity, is-a, has-a, and event.

[0062] WHERE represents a constraint.

[0063] The constraint operators only consider three types: "=", "AND", and "OR".

[0064] For detailed instructions on how to write prompt, please refer to the examples in the schema generation module and information extraction module.

[0065] Pre-trained language models:

[0066] We collect a large amount of unlabeled text corpus from the Internet, and first conduct self-supervised training on the text corpus based on the transformer encoder-decoder deep learning architecture to obtain a pre-trained language model. Then, we improve the performance of the schema generation module and the information extraction module through corresponding fine-tuning.

[0067] Such as masked language model (MLM) or causal language model (CLM);

[0068] Furthermore, the loss function of the pre-trained language model is:

[0069] L text =∑ x -log p(x target |x;θ e θ d )……(1)

[0070] In equation (1), θ e θ d These are the parameters for the encoder and decoder, respectively; x is the input text, x target For the text to be predicted, if a masked language model is used, x target That is, the text that is obscured;

[0071] If a causal language model is used, x target That is, the next word, the next paragraph, or the next sentence.

[0072] schema generation module

[0073] The input format for the schema generation module is: SQL[text] text;

[0074] Here, [text] is a special word (token) to distinguish the SQL statement from the original text; the output of the schema generation module is a schema fragment, which uses a sequence to represent the specified part of the corresponding SELECT.

[0075] The encoder part of the schema generation module uses the same encoder as the pre-trained language model, while the decoder part adopts a generative approach. The loss function is:

[0076] L schema =∑ (x，schema) -log p(schema|x, metaSQL; θ e θ d )……(2)

[0077] Where, θ e θ d These are the parameters for the encoder and decoder, respectively; x is the original text; metaSQL is the meta information expressed in SQL language; schema is the schema information to be output.

[0078] In detail, regarding the specific syntax of SQL,

[0079] (1) If it is a specific entity type

[0080] The SQL syntax is SELECT type FROM entity, or SELECT entity.type.

[0081] The generated result is [{entity.type:#},...], where [] indicates that there may be multiple results;

[0082] (2) If the is-a relationship is extracted based on a known entity type, the SQL syntax is SELECT type FROM is-a WHERE entity.type = #, and the generated result is [{entity.type:#,is-a.type:#},...];

[0083] (3) If it is to extract the specific schema of the entity and the relation, the SQL syntax is SELECT entity.type,is-a.type, and the generated result is also [{entity.type:#,is-a.type:#},...];

[0084] (4) If it is to extract the specific schema of the entity and the event together, the SQL is written as SELECT who.type,what.type,why.type,when.type,where.type,how.type FROM event, and the generated result is [{who.type:#,what.type:#,why.type:#,when.type:#,where.type:#,how.type:#},...];

[0085] (5) If the event type is extracted first and then the event schema is extracted.

[0086] It can be divided into two steps: SELECT what.type FROM event and SELECT who.type,why.type,when.type,where.type,how.type FROM event WHERE what.type=#,

[0087] The generated results are [{what.type:#},...] and [{who.type:#,what.type:#,why.type:#,when.type:#,where.type:#,how.type:#},...] respectively.

[0088] Information extraction module

[0089] like Figure 3 As shown, the information extraction module adopts the same model architecture as the schema generation module.

[0090] The input for information extraction is SQL[text] text, and the output is structured data information, corresponding to the specified part of the SELECT statement.

[0091] Regarding the specific syntax of SQL,

[0092] If it involves jointly extracting entity and is-a relationship information...

[0093] The SQL writing method is SELECT entity.value,is-a.value WHERE entity.type=#ANDis-a.type=#,

[0094] The extraction result is [{entity.type:#,entity.value:#,is-a.type:#,is-a.value:#},...], where [] indicates that there may be multiple results;

[0095] If entity and event specific information is jointly extracted, the SQL writing method is SELECT who.value,what.value,why.value,when.value,where.value,how.value FROM event WHEREwho.type=#AND what.type=#AND why.type=#AND when.type=#AND where.type=#AND how.type=#, the generated result is [{who.type:#,what.type:#,why.type:#,when.type:#,where.type:#,how.type:#,who.value:#,what.value:#,why.value:#,when.value:#,where.value:#,how.value:#},...].

[0096] It should be noted that using SQL makes it easier to incorporate background or contextual information beyond the original text. When extracting event information, entity information is extracted first, and then relevant background fragments are retrieved from the constructed knowledge graph based on the entity information.

[0097] The corresponding SQL syntax for event extraction is: SELECT what.value, why.value, when.value, where.value, how.value FROM event WHERE who.type=#AND who.value=#ANDwhat.type=#AND why.type=#AND when.type=#AND where.type=#AND how.type=#AND who.info=#. Here, who.info can retrieve other information about related entities from the constructed knowledge graph based on who.type and who.value, and then organize it into text and store it in who.info.

[0098] The above method only demonstrates the simplest way to express background information, but it is not limited to the expression of who.info. It uses a more structured way to express background information.

[0099] The loss function of the information extraction module is:

[0100] L data =∑ (x，data) -log p(data|x,schemaSQL;θ e θ d )……(3)

[0101] In its formula (3), θ e θ d These are the parameters for the encoder and decoder, respectively; x is the original text; schemaSQL is the schema information expressed in SQL language; and data is the structured data information to be output.

[0102] Information fusion module

[0103] When merging structured data fragments extracted from a single unstructured text into the original knowledge graph, it is necessary to verify the credibility of the information.

[0104] The validation formula for the value of a key field in a schema is as follows:

[0105] value = max k (∑ jweight j (value k (), for k=1,...N)……(4)

[0106] In equation (4), weight j The credibility of source j is calculated using the PageRank method; k = 1,...,N represents that value has N possible values. k Represents the k-th value; the same value extracted from different sources j. k Each value has a different weight. After summing the weight values, the value with the highest corresponding score is taken. k That is, the value corresponding to the key.

[0107] When structured data is integrated into a knowledge graph, the source, original text, generated schema, and extracted structured data are saved in another database.

[0108] The validated data can be continuously used as training data to update the schema generation module and information extraction module.

[0109] Structured data integrated into knowledge graphs is not 100% reliable. As new information is added, older knowledge may no longer maintain its highest ranking and will be updated. The training data will also be updated accordingly, continuously improving the reliability of the extraction model.

[0110] Building upon the modules described above, further joint training can be performed when continuously training the schema generation module and the information extraction module (i.e., the schema generation module and the information extraction module). The loss function is as follows:

[0111] L = L text +L schema +L data ……(5)

[0112] This embodiment also provides a working method for an automatic knowledge graph construction system, including the following steps:

[0113] S1: Pre-train the language model using a self-supervised approach, as enumerated by the pre-trained language model. The process is as follows: Figure 2 As shown, the unstructured text corpus can be collected from various unlabeled Internet text corpora;

[0114] S2: Collect unstructured text and its corresponding schema information as training data for the schema generation module. The cold start method for training data involves collecting structured knowledge and corresponding text corpora from encyclopedia-like web pages. Specifically, one method is to search for text containing structured values ​​using structured knowledge, and then use this text and the schema of the structured knowledge as training data. Training is performed using the methods listed for the schema generation module; for example... Figure 3 As shown

[0115] S3: Take the unstructured text and its corresponding schema information as input, and the specific data values ​​corresponding to the text and schema as output, to train the information extraction module, such as... Figure 4 As shown. The training data used for information extraction can be obtained in the same way as S2, the difference being that the original text, structured schema, and structured data need to be saved simultaneously. The model is trained using the training methods listed in the information extraction module;

[0116] S4: After obtaining the schema generation module through S2 and the information extraction module through S3, a complete information extraction can be performed on a new document: First, the schema information is obtained through the schema generation module, and then structured data information is obtained based on the text and schema information; for example... Figure 5 As shown;

[0117] S5: After obtaining structured data information from any source through S4, the structured data is verified and fused using the knowledge fusion methods listed in the information fusion module.

[0118] In this process, when knowledge of "market transactions" from different sources conflicts, knowledge with lower credibility is discarded, while knowledge with higher credibility is integrated into the knowledge graph.

[0119] The validated raw text and structured data are added to the training data, and the training continues using the training methods of the schema generation module and the information extraction module.

[0120] The specific verification process is as follows: Figure 6 As shown, see the example of the fusion output result. Figure 7-10 .

[0121] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An automatic knowledge graph construction system, characterized in that, Includes information collection module, schema generation module, The system includes an information extraction module, an information fusion module, and a pre-trained language model. The schema generation module and the information extraction module use a deep learning model that provides a prompt to represent natural language text. The information collection module is used to collect unstructured text corpora and their corresponding schema information, including but not limited to unlabeled internet text corpora. The schema generation module consists of the same encoder and generative decoder as the pre-trained language model. The schema generation module is used to generate schema fragments represented as sequences, corresponding to the specified part of SELECT. The information extraction module is used to extract natural language corpus and output structured information. Unstructured text and its corresponding schema information are concatenated as input, and the specific data values ​​corresponding to the text and schema are output. The structured information is expressed in SQL. The pre-trained language model is set up on the server and trained using a deep learning architecture based on transformer encoder-decoder. It performs self-supervised training on text corpora and then feeds back to the schema generation module and information extraction module after appropriate fine-tuning to improve training accuracy. The information fusion module is used to extract structured data containing partial schema information from a single unstructured text. When fusing fragments of structured data into a knowledge graph, the credibility of the information is verified.

2. The automatic knowledge graph construction system according to claim 1, characterized in that, The loss function between the encoder and decoder in the schema generation module is expressed as follows: ； in, , These are the parameters for the encoder and decoder, respectively; The first line contains the original text; the second line contains the meta information expressed in SQL; and the third line contains the schema information to be output.

3. The automatic knowledge graph construction system according to claim 1, characterized in that, The loss function of the information extraction module is: ； in, , These are the parameters for the encoder and decoder, respectively; x is the original text; schemaSQL is the schema information expressed in SQL language; and data is the structured data information to be output.

4. The automatic knowledge graph construction system according to claim 1, characterized in that, The loss function of the pre-trained language model is: ； in, , These are the parameters for the encoder and decoder, respectively; x is the input text. For the text to be predicted, if a masked language model is used, That is, the obscured text; if a causal language model is used, That is, the next word, the next paragraph, or the next sentence.

5. The automatic knowledge graph construction system according to claim 1, characterized in that: The information fusion module uses the following formula to verify the value of a certain attribute field key in the schema: ； in, The credibility of source j is represented by the PageRank algorithm, which is used to calculate it; k = 1, ..., N represents that value has N possible values. Represents the k-th value; the same value extracted from different sources j. Different weight values ​​are assigned to different scores. After summing the weight values, the score with the highest corresponding score is selected. That is, the value corresponding to the key.

6. The working method of the automatic knowledge graph construction system according to any one of claims 1-5, characterized in that, Includes the following steps: S1: Pre-train the language model using the self-supervised methods listed in the pre-trained language model enumeration, where the unstructured text corpus includes various unlabeled Internet text corpora. S2: Collect unstructured text and its corresponding schema information as training data for the schema generation module; search for text containing structured values ​​through structured knowledge, and use this text and the schema of structured knowledge as training data. S3: The unstructured text and its corresponding schema information are concatenated as input, and the specific data values ​​corresponding to the text and schema are output as the training of the information extraction module; and the specific output data format is given. S4: After obtaining the schema generation module through S2 and the information extraction module through S3, perform complete information extraction on a new document: First, obtain schema information through the schema generation module, and then obtain structured data information based on the text and schema information; S5: After obtaining structured data information from any source through S4, the structured data is verified and fused using the knowledge fusion methods listed in the information fusion module; the verified original text and structured data are added to the training data, and continuous training is carried out through the training methods of the schema generation module and the information extraction module.

7. The working method of the automatic knowledge graph construction system according to claim 6, characterized in that, The training data used by the information extraction module in step S2 needs to simultaneously save the original text, the structured schema, and the structured data.

Images