Method for constructing knowledge graph based on ie-triple

Abstract

A knowledge graph construction method based on IE-Triple, the method comprises the following steps: step 1, given original text and triple pattern guide structure; step 2, the triple pattern structure guide is embedded in the original text through the triple pattern guide structure, and the triple pattern structure guide is used as the input of the model; step 3, the triple pattern structure generator carries out information extraction on the original text in the input, and generates the corresponding mode structure triple according to the triple guide structure; step 4, the IE-Triple model pre-training and fine-tuning strategy is provided, so that the model can adapt to the downstream task. For users, users only need to customize the triple pattern guide structure, and triple extraction can be realized; at the same time, through the pre-training and fine-tuning method designed by the application, the IE-Triple model is suitable for triple extraction tasks in various scenes.

Description

Technical Field

[0001] This invention relates to the fields of knowledge graph construction and deep learning technology, specifically to a knowledge graph construction method based on IE-Triple. Technical Background

[0002] Knowledge graphs were a concept proposed by Google in 2012 and officially announced at the Google I / O conference that May. The company uses technologies such as knowledge representation, knowledge extraction, and knowledge fusion to extract and construct knowledge graphs from massive amounts of data. These graphs contain billions of entities and trillions of relationships, covering knowledge across various fields, including people, places, organizations, events, technology, and art. Google's knowledge graphs are applied to various products, including search engines, to improve the quality and accuracy of search results and provide users with a better search and user experience. The emergence of knowledge graphs represents a significant advancement in fields such as artificial intelligence and natural language processing, and has been a hot topic in the field of AI in recent years. With continuous technological advancements, knowledge graphs will become one of the important foundations and core technologies in the field of artificial intelligence.

[0003] The quality of a knowledge graph directly impacts the user experience of an application. Many companies invest significant effort in building excellent knowledge graphs, such as DBpedia used by Wikipedia, Freebase (an open knowledge base with user-added and reviewed content), and WordNet (a large semantic network containing words from over 20 languages). When a high-quality knowledge graph is constructed, the knowledge graph construction methodology plays a crucial role.

[0004] Knowledge graph construction methods refer to the techniques of integrating large amounts of distributed heterogeneous data resources, extracting information such as entities, relationships, and attributes, and presenting this information in the form of triples. A triple consists of a head entity, a tail entity, and a relationship, where the head entity represents the subject, the tail entity represents the object, and the relationship represents the connection between the subject and object. Named entity recognition (Named Entity Recognition) and relationship prediction are core problems in knowledge graph construction methods. Entity and relationship recognition can be achieved through techniques such as machine learning, rule matching, and template-based methods. Among these, machine learning is one of the most widely used methods, enabling automatic entity and relationship recognition through model training. Existing knowledge graph construction methods fall into two categories: pipeline models, where named entity recognition and relationship prediction are performed sequentially; and joint models, which are end-to-end models that directly extract triples from unstructured language. Among them, end-to-end models can directly extract triples from the original text and have advantages such as lower model calling costs and stronger adaptability, and are gradually becoming the mainstream knowledge graph construction model.

[0005] Common knowledge graph construction methods mainly include rule-based methods and encoder-decoder-based methods. Rule-based methods typically extract triples based on manually created rule-matching patterns. These rules can be linguistic or semantic, such as those used in syntactic analysis and named entity recognition. However, these methods require manual rule construction and struggle to cover all linguistic phenomena and domain knowledge, thus limiting their scalability and applicability. Encoder-decoder-based methods use neural network models to learn from labeled text, constructing a triple extraction model, and then applying this model to extract triples from the original text. The advantage of this method is its ability to automatically learn features and adjust weights and biases, demonstrating excellent generalization ability across different tasks, thus attracting significant attention and research from scholars in the field. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies and to extract triples from raw text for knowledge graph construction, this invention provides a knowledge graph construction method based on IE-Triple. It designs a guided pattern structure suitable for triple extraction tasks, embedding the original text into the guided pattern structure, and using an encoder-decoder pair to generate triples matching the target structure. This leads to the design of a knowledge graph construction method based on IE-Triple (Information Extraction to Triple). Furthermore, through pre-training and fine-tuning of the IE-Triple model, it is made applicable to triple extraction tasks in various scenarios.

[0007] The technical solution adopted in this invention is as follows:

[0008] A knowledge graph construction method based on IE-Triple includes the following steps:

[0009] Step 1: Given the original text and the triplet pattern guide structure;

[0010] Step 2: The triplet pattern structure guide embeds a triplet pattern guide structure into the original text and uses it as input to the model.

[0011] Step 3: The triplet pattern structure generator extracts information from the original text in the input and generates triplets with corresponding pattern structures based on the triplet guided structure transformation.

[0012] Step 4: Provide IE-Triple model pre-training and fine-tuning strategies to enable the model to adapt to downstream tasks.

[0013] Steps 1-3 complete the construction of the IE-Triple, and step 4 performs pre-training and fine-tuning of the IE-Triple.

[0014] Furthermore, the process of step 1 is as follows:

[0015] 1.1 Given the original text w = [w1, w2, w...] i ,…]. Among them, w i Let be the vector of the i-th word.

[0016] 1.2 This invention proposes a Triple-schema Instructed Struct (TIS) to indicate the entity type and relation type associated with the target information, defined as follows:

[0017] p=[[head],EntityType,[Relation],RelationType,[tail],EntityType]

[0018] Where p is the triplet schema guide structure, the markers [head], [Relation] and [tail] represent the location information of the head entity, relation and tail entity in TIS, and the markers EntityType and RelationType specify the type of the corresponding entity and relation.

[0019] Furthermore, the process of step 2 is as follows:

[0020] For the original text w, the Triple-schema StructuralInstructor (TSI) embeds TIS as a prefix into w and uses it as input to the IE-Triple model, as shown in the following formula:

[0021]

[0022] in, This represents the value calculated by TSI. This indicates a vector concatenation operation, where [SEP] represents the separation information between w and TIS;

[0023] As described in steps 1 and 2, TSI is a pattern-structure-based prompting mechanism that effectively guides the IE-Triple model to generate corresponding guided triples. This mechanism enables the model to adaptively control and recognize head entities, tail entities, and the relationships between them, as well as share semantic knowledge across different labels and tasks.

[0024] Furthermore, the process of step 3 is as follows:

[0025] 3.1 To generate valid triples from the IE-Triple model, a Triple-schema Generation Struct (TGS) needs to be defined. The TGS consists of two parts: 1) Entity Part: Predefined entities identified from the original text, including head and tail entities; 2) Relation Part: Predefined relationships identified from the original text, including relationships between head and tail entities and relationships between entities and attributes.

[0026] Each TGS contains three types of tokens: 1) EntityType*: indicating the specified type of the head and tail entities in the triple; 2) HeadName / TailName: indicating the actual value of the head or tail entity extracted from the original text; 3) RelationType*: indicating the specified type of the relation in the triple. Furthermore, the colon “:” in the TGS indicates the mapping from the specified type of the entity to its actual value, and the two structure indicators “{” and “}” are used to encompass all the information of a single triple.

[0027] 3.2. The value calculated by TSI is used as the model input, and the target triples are extracted by TSG.

[0028] The process described in section 3.2 is as follows:

[0029] 3.2.1 The encoder first performs positional encoding on the input vector, marking each position of the input vector. This involves superimposing the positional encoding onto the original input vector, resulting in an encoded vector with the same dimension as the input vector. The formula is as follows:

[0030]

[0031] in, Represents the encoded vector. X represents the value obtained from step 2. pos Indicates positional encoding;

[0032] 3.2.2 The multi-head attention mechanism enhances the ability to adaptively recognize triple information in the original text. The encoder consists of multiple independent attention heads, which are implemented based on the ScaledDot-Product Attention mechanism. ScaledDot-Product Attention is an adaptive weighted attention mechanism, described as a mapping from the input vector to the hidden state through the query vector, key vector, and value vector. The query vector, key vector, and value vector are generated as a linear combination of the input vector, as shown in the following formula:

[0033]

[0034]

[0035]

[0036] Where Q is the query vector, K is the key vector, and V is the value vector. For the encoding vector in step 3.2.1, W Q W K W V This is the weight matrix;

[0037] 3.2.3 The vector output by the attention mechanism is essentially a weighted combination of value vectors. This weight matrix is ​​generated by comparing the query vector and the key vector, reflecting the proportion of words in the original text related to the triple information. Furthermore, the attention head is divided equally according to the input dimension, denoted as h parts. For PW h Let h be the hidden state generated by the i-th attention head. i The formula is as follows:

[0038]

[0039] Where Softmax(·) represents the normalization exponential function, W Q W K W V The weight matrix d in step 3.2.2 is... model Represents the input vector PW h The dimension of the matrix is ​​T, where T represents the matrix transpose.

[0040] 3.2.4. The h output by each attention head in step 3.2.3 i The process involves splicing, followed by residual connections and hidden layer regularization. The residual connections are part of step 3.2.1. and h i The linear superposition and residual connections are used to address gradient vanishing and gradient explosion problems that arise with increasing network depth. Hidden layer regularization normalizes the hidden layers in the network to a standard normal distribution. Then, the output is passed through a bit-by-bit feedforward network to obtain the encoder's hidden state, as shown in the following formula:

[0041]

[0042] Where H represents the encoder hidden state, FFN(·) represents the feedforward neural network, and LayerNorm(·) represents the hidden layer regularization function;

[0043] 3.2.5 The components of the decoder are similar to those of the encoder, but the difference is that the position-encoded vectors undergo masked multi-head attention computation. The masking operation is to preserve autoregressive properties, that is, to ensure that the prediction result depends only on the generated output words. The masked multi-head attention mechanism, residual connections, and hidden layer regularization are considered as the front end of the decoder, denoted as Decoder. f (·), the calculation formula is as follows:

[0044]

[0045] Where, d i Let d represent the i-th word segmentation vector.i in[d1,d2,d i [,...] represents the i-th operation of the loop, LayerNorm(·) represents the hidden layer regularization function, and Attention... mask (·)Will The vectors are masked and then processed using a multi-head attention mechanism. d i The vector generated by position encoding. Represents the query vector, key vector, and value vector that participate in the mask multi-head attention computation;

[0046] 3.2.6 The back-end structure of the decoder is the same as that of the encoder, sequentially employing a multi-head attention mechanism, residual connections and hidden layer regularization, a bit-by-bit feedforward network, and residual connections and hidden layer regularization. The input to the back-end part comes from the hidden state of the encoder in step 3.2.4 and the vector generated by the front-end part of the decoder in step 3.2.5. The back-end part of the decoder is denoted as Decoder. b Finally, the vector output from the backend is processed by a fully connected layer and a softmax function to obtain the predicted probability y. i ,d i The calculation formula is as follows:

[0047] y i ,d i =Softmax FC (Decoder b (H;Decoder f ([d1,d2,…d D ])))

[0048] Among them, y i ,d i d i The matching probability with the i-th token in TGS, Softmax FC (·) represents a fully connected layer based on the Softmax function, where H represents the hidden state from the encoder in step 3.2.4. f (·) indicates the result calculated in step 3.2.5. Finally, the content of the TGS expression generated by the model prediction is output in text form.

[0049] Step 4 describes how to pre-train and fine-tune the IE-Triple model. First, large datasets are collected from the internet and preprocessed, including structured data, semi-structured databases, and unstructured data. These samples are then processed into text-to-TSG (text-to-TSG) structures and used as pre-training samples for the model. Structured data can be converted based on its relational patterns, while unstructured data requires manual conversion. To ensure the validity of unstructured data, public knowledge bases provided by Wikipedia and Baidu Baike are used. Next, the IE-Triple model is pre-trained uniformly using the aforementioned datasets. Finally, the pre-trained IE-Triple model is adapted to specific downstream tasks based on the needs of real-world scenarios. Since the pre-trained IE-Triple model is based on a general knowledge base, it has the ability to share knowledge in triple extraction tasks across different scenarios.

[0050] The process of step 4 is as follows:

[0051] 4.1 This invention divides the pre-training of the IE-Triple model into three parts: 1) pre-training of the encoder part of the IE-Triple model; 2) pre-training of the decoder part of the IE-Triple model; 3) pre-training of the IE-Triple model as a whole. To this end, a loss function is defined for each of the above three pre-training tasks.

[0052] 4.2 For on-demand fine-tuning, given a pre-trained IE-Triple model, fine-tuning is used to allow the model to quickly adapt to triple extraction for specific scenarios. Given a labeled corpus, the cross-entropy loss function is used to fine-tune the IE-Triple model. The loss function for the fine-tuning stage is defined as follows:

[0053]

[0054] in, This represents the loss function during the fine-tuning phase.<t′,s′,y′> Let y' represent the sample to be tested, t' represent the original text to be tested, s' represent the guided triplet structure to be tested, and D represent the original text to be tested. task Let represent a corpus for a specific scenario, D represent the complete set of samples in the pre-training process, and p(·) and q(·) are both conditional probability distributions. and To adjust the parameters.

[0055] Furthermore, the process of step 4.1 is as follows:

[0056] 4.1.1. For the pre-training of the IE-Triple encoder, in order for the encoder part to capture the mapping ability from the original text to the hidden layer state, that is, whether there is information related to the triple-guided structure in the original text. Therefore, the data parts involved are the research object text data t and the triple-guided structure s, and the present invention regards these two types of data as a data pair <t, s>. During the model training process, if only positive example samples are provided to the model, the model will only remember the correct triples in the pre-training data. Therefore, negative example samples need to be constructed by random matching, and the pre-training objective loss function of the IE-Triple encoder is given as follows:

[0057]

[0058] Among them, represents the objective loss function of the encoder pre-training, y e represents the output of the model in this pre-training, is the positive example sample of the encoder pre-training, is the negative example sample of the encoder pre-training, log(·) is the logarithmic function, p(·) is the conditional probability distribution, is the parameter of the encoder.

[0059] 4.1.2. For the pre-training of the IE-Triple decoder, the role of the decoder is to parse the hidden state generated by the encoder into the semantic representation of the target triple. By changing the triple pattern generation structure to create negative example samples, the ability of the decoder to effectively identify positive example samples is pre-trained. Therefore, the pre-training objective loss function of the IE-Triple decoder is given:

[0060]

[0061] Among them, represents the objective loss function of the decoder pre-training, y d represents the data to be measured, represents the i-th token of y d y d <i represents the substring of the data to be measured with a length of i, is the positive example sample of the decoder pre-training, is the negative example sample of the decoder pre-training, log(·) is the logarithmic function, p(·) is the conditional probability distribution, is the parameter of the encoder;

[0062] 4.1.3 For the overall pre-training of the IE-Triple model, to effectively train the model's extraction capabilities, training samples are constructed using a text-to-target triplet structure. Formally, this part is considered an unsupervised masking language model task, masking a portion of the words in the original text and generating the masked portion. Since the model's input already contains the output content, it's necessary to rearrange the word order of the marked words in the original text and the word order of the triplet pattern generation structure markers. The loss function for the overall pre-training of the IE-Triple model is given below.

[0063]

[0064] in, Let y represent the overall pre-training target loss function, y represent the triplet pattern generation structure, and t represent the original text. Generate a structure for the triplet pattern after word order rearrangement. Let D be the original text after word order rearrangement, and p(·) be the conditional probability distribution. These represent the training parameters in steps 4.1.1 and 4.1.2, respectively.

[0065] 4.1.4 Based on the objectives of the three types of pre-training tasks mentioned above, the final objective loss function is a combination of the objective loss functions of the above tasks:

[0066]

[0067] in, This represents the final pre-training loss function. The loss function derived from steps 4.1.1, 4.1.2, and 4.1.3.

[0068] The beneficial effects of this invention are: through the triplet pattern guidance structure and generation structure designed in this invention, and under the translation action of the encoder-decoder, the IE-Triple model can directly extract valid triples from the original text; for users, they only need to customize the triplet pattern guidance structure to achieve triplet extraction; at the same time, through the pre-training and fine-tuning methods designed in this invention, the IE-Triple model is applicable to triplet extraction tasks in various scenarios.

[0069] Instruction manual illustrations

[0070] Figure 1 A diagram of the IE-Triple model was presented.

[0071] Figure 2 A diagram showing the structure for generating triplet patterns was presented.

[0072] Figure 3An encoder-decoder diagram is presented.

[0073] Figure 4 The statistical data of the IE-Triple model experimental dataset is presented in a graph.

[0074] Figure 5 A diagram of the experimental environment was shown.

[0075] Figure 6 A comparison chart of Precision, Recall, and F1 evaluation metrics was presented.

[0076] Figure 7 A comparison chart showing the effects of the fine-tuning was presented. Detailed Implementation

[0077] The present invention will now be further described with reference to the accompanying drawings.

[0078] like Figure 1 As shown, given original text and a specific triplet pattern guidance structure, the IE-Triple model generates target triples. From the overall structure of the IE-Triple model, it can be divided into two parts: a triplet pattern guidance structure guide and a triplet pattern structure generator. The triplet pattern structure guide embeds the triplet pattern guidance structure into the original text and uses it as input to the model. The triplet pattern structure generator extracts information from the original text based on an encoder-decoder and transforms it according to the triplet guidance structure to generate the corresponding target triples.

[0079] Step 1: Given the original text and the triplet pattern guide structure;

[0080] The process of step 1 is as follows:

[0081] 1.1 Given the original text w = [w1, w2, w...] i ,…]. Among them, w i Let i be the vector of the i-th word;

[0082] 1.2 This invention proposes a Triple-schema Instructed Struct (TIS) to indicate the entity type and relation type associated with the target information, defined as follows:

[0083] p=[[head],EntityType,[Relation],RelationType,[tail],EntityType]

[0084] Where p is the triplet schema bootstrap structure, the markers [head], [Relation] and [tail] represent the location information of the head entity, relation and tail entity in TIS, and the markers EntityType and RelationType specify the type of the corresponding entity and relation;

[0085] Step 2: The triplet pattern structure guide embeds a triplet pattern guide structure into the original text and uses it as input to the model;

[0086] The process of step 2 is as follows:

[0087] For the original text w, the Triple-schema StructuralInstructor (TSI) embeds TIS as a prefix into w and uses it as input to the IE-Triple model, as shown in the following formula:

[0088]

[0089] in, This represents the value calculated by TSI. This indicates a vector concatenation operation, where [SEP] represents the separation information between w and TIS;

[0090] Based on the descriptions in steps 1 and 2, TSI is a pattern-structure-based prompting mechanism that can effectively guide the IE-Triple model to generate corresponding guided triples. This mechanism enables the model to adaptively control and recognize head entities, tail entities, and the relationships between them, as well as share semantic knowledge from different labels and tasks.

[0091] Step 3: The triplet pattern structure generator extracts information from the original text in the input and generates triplets with corresponding pattern structures based on the triplet guided structure transformation.

[0092] The process of step 3 is as follows:

[0093] 3.1 To generate valid triples from the IE-Triple model, a Triple-schema Generation Struct (TGS) needs to be defined, such as... Figure 2 As shown, TGS consists of two parts: 1) EntityPart: Predefined entities identified from the original text, including head entities and tail entities; 2) RelationPart: Predefined relationships identified from the original text, including the relationship between head and tail entities and the relationship between entities and attributes.

[0094] Each TGS contains three types of tokens: 1) EntityType*: indicating the specified type of the head and tail entities in the triple; 2) HeadName / TailName: indicating the actual value of the head or tail entity extracted from the original text; 3) RelationType*: indicating the specified type of the relation in the triple. Furthermore, the colon ":" in the TGS represents the mapping from the specified type of the entity to the actual value, and the two structure indicators "{" and "}" are used to encompass all the information of a single triple. The TGS designed in this invention adaptively matches and generates the corresponding triple structure based on the triple pattern structure guide. For example, given the sentence "West Lake is a 5A scenic spot in Hangzhou's West Lake District," three triples with different patterns can be generated: 1) "{ScenicSpots: West Lake, inCity,City:Hangzhou}"; 2) "{ScenicSpots: West Lake, inDistrict,District:West Lake District}"; 3) "{ScenicSpot:West Lake,Ranks,RanksLevel:5A}".

[0095] 3.2. Using the values ​​calculated by TSI as model input, target triples are extracted using TSG;

[0096] The process described in section 3.2 is as follows:

[0097] 3.2.1 The encoder first performs positional encoding on the input vector, marking each position of the input vector. This involves superimposing the positional encoding onto the original input vector, resulting in an encoded vector with the same dimension as the input vector. The formula is as follows:

[0098]

[0099] in, Represents the encoded vector. X represents the value obtained from step 2. pos Indicates positional encoding;

[0100] 3.2.2 Multi-head attention mechanisms enhance the ability to adaptively recognize triple information in the original text. The encoder consists of multiple independent attention heads, implemented using ScaledDot-Product Attention, an adaptive weighted attention mechanism. ScaledDot-Product Attention is described as a mapping from the input vector to the hidden state through query vectors, key vectors, and value vectors. The query vector, key vector, and value vector are generated as a linear combination of the input vectors, as shown in the following formula:

[0101]

[0102]

[0103]

[0104] Where Q is the query vector, K is the key vector, and V is the value vector. For the encoding vector in step 3.2.1, W Q W K W V This is the weight matrix;

[0105] 3.2.3 The vector output by the attention mechanism is essentially a weighted combination of value vectors. This weight matrix is ​​generated by comparing the query vector and the key vector, reflecting the proportion of words in the original text related to the triple information. Furthermore, the attention head is divided equally according to the input dimension, denoted as h parts. For PW h Let h be the hidden state generated by the i-th attention head. i The formula is as follows:

[0106]

[0107] Where Softmax(·) represents the normalization exponential function, W Q W K W V The weight matrix d in step 3.2.2 is... model Represents the input vector PW h The dimension of the matrix is ​​T, where T represents the matrix transpose.

[0108] 3.2.4. The h output by each attention head in step 3.2.3 i Perform a splicing operation, followed by residual connections and hidden layer regularization (corresponding to respectively). Figure 3 (The "addition" and "normalization" in the text), where residual connection is the step in step 3.2.1. and h i The linear superposition and residual connections are used to address gradient vanishing and gradient explosion problems that arise with increasing network depth. Hidden layer regularization normalizes the hidden layers in the network to a standard normal distribution. Then, the output is passed through a bit-by-bit feedforward network to obtain the encoder's hidden state, as shown in the following formula:

[0109]

[0110] Where H represents the encoder hidden state, FFN(·) represents the feedforward neural network, and LayerNorm(·) represents the hidden layer regularization function;

[0111] 3.2.5 The components of the decoder are similar to those of the encoder, but the difference is that the position-encoded vectors undergo masked multi-head attention computation. The masking operation is to preserve autoregressive properties, that is, to ensure that the prediction result depends only on the generated output words. The masked multi-head attention mechanism, residual connections, and hidden layer regularization are considered as the front end of the decoder, denoted as Decoder. f (·), the calculation formula is as follows:

[0112]

[0113] Where, d i Let d represent the i-th word segmentation vector. i in[d1,d2,d i [,...] represents the i-th operation of the loop, LayerNorm(·) represents the hidden layer regularization function, and Attention... mask (·) is for general The vectors are masked and then processed using a multi-head attention mechanism. d i The vector generated by position encoding. Represents the query vector, key vector, and value vector that participate in the mask multi-head attention computation;

[0114] 3.2.6 The back-end structure of the decoder is the same as that of the encoder, sequentially employing a multi-head attention mechanism, residual connections and hidden layer regularization, a bit-by-bit feedforward network, and residual connections and hidden layer regularization. The input to the back-end part comes from the hidden state of the encoder in step 3.2.4 and the vector generated by the front-end part of the decoder in step 3.2.5. The back-end part of the decoder is denoted as Decoder. b (·). Finally, the vector output from the backend is processed by a fully connected layer and a Softmax function to obtain the predicted probability y in step 3.2. i ,d i The calculation formula is as follows:

[0115] y i ,d i =Softmax FC (Decoder b (H;Decoder f ([d1,d2,…d D ])))

[0116] Among them, y i ,d i d i The matching probability with the i-th token in TGS, Softmax FC(·) represents a fully connected layer based on the Softmax function, where H represents the hidden state from the encoder in step 3.2.4. f (·) indicates the result calculated in step 3.2.5. Finally, the content of the TGS expression generated by the model prediction is output in text form;

[0117] Step 4: Provide IE-Triple model pre-training and fine-tuning strategies to enable the model to adapt to specific downstream tasks;

[0118] In step 4, this invention describes how to pre-train and fine-tune the IE-Triple model. First, a large dataset is collected from the internet and preprocessed, including structured data, semi-structured databases, and unstructured data. These samples are processed into text-to-TSG (text-to-TSG) patterns and used as pre-training samples for the model. Structured data can be converted based on its relational patterns, while unstructured data requires manual conversion. To ensure the validity of unstructured data, public knowledge bases provided by Wikipedia and Baidu Baike are used. Then, the IE-Triple model is uniformly pre-trained using the aforementioned dataset. Finally, the pre-trained IE-Triple model is adapted to specific downstream tasks based on the needs of real-world scenarios. Since the pre-trained IE-Triple model is based on a general knowledge base, it possesses the ability to share knowledge in triple extraction tasks across different scenarios.

[0119] The process of step 4 is as follows:

[0120] 4.1 This invention divides the pre-training of the IE-Triple model into three parts: 1) pre-training of the encoder part of the IE-Triple model; 2) pre-training of the decoder part of the IE-Triple model; 3) pre-training of the IE-Triple model as a whole. For this purpose, a loss function is defined for each of the above three pre-training tasks.

[0121] Furthermore, the process in section 4.1 is as follows:

[0122] 4.1.1. For the pre-training of the IE-Triple encoder, in order to enable the encoder part to capture the mapping ability from the original text to the hidden layer state, that is, whether there is information related to the triple guiding structure in the original text. Therefore, the data parts involved are the research object text data t and the triple guiding structure s. In this invention, these two types of data are regarded as the data pair <t, s>. During the model training process, if only positive example samples are provided to the model, the model will only remember the correct triples in the pre-training data. Therefore, negative example samples need to be constructed by random matching, and the pre-training objective loss function of the IE-Triple encoder is given as follows:

[0123]

[0124] Among them, represents the pre-training objective loss function of the encoder, y e represents the output of the model in this pre-training, is the positive example sample for the encoder pre-training, is the negative example sample for the encoder pre-training, log(·) is the logarithmic function, p(·) is the conditional probability distribution, are the parameters of the encoder;

[0125] 4.1.2. For the pre-training of the IE-Triple decoder, the role of the decoder is to parse the hidden state generated by the encoder into the semantic representation of the target triple. By changing the triple pattern generation structure to create negative example samples, the ability of the decoder to effectively identify positive example samples is pre-trained. Therefore, the pre-training objective loss function of the IE-Triple decoder is given:

[0126]

[0127] Among them, represents the pre-training objective loss function of the decoder, y d represents the data to be tested, represents the i-th token of y d y d <i represents the substring of the data to be tested with a length of i, is the positive example sample for the decoder pre-training, is the negative example sample for the decoder pre-training, log(·) is the logarithmic function, p(·) is the conditional probability distribution, are the parameters of the encoder;

[0128] 4.1.3 For the overall pre-training of the IE-Triple model, in order to effectively train the extraction capability of the entire model, training samples are constructed in the form of text-to-target triplet structures. Formally, this part is regarded as an unsupervised masking language model task, masking a portion of the words in the original text and generating the masked part. Since the input part of the model already contains the output content, it is necessary to rearrange the word order of the marked words in the original text and the word order of the triplet pattern generation structure markers. The loss function for the overall pre-training of the IE-Triple model is given:

[0129]

[0130] in, Let y represent the overall pre-training target loss function, y represent the triplet pattern generation structure, and t represent the original text. Generate a structure for the triplet pattern after word order rearrangement. Let D be the original text after word order rearrangement, and p(·) be the conditional probability distribution. These represent the training parameters in steps 4.1.1 and 4.1.2, respectively.

[0131] 4.1.4 Based on the objectives of the three types of pre-training tasks mentioned above, the final objective loss function is a combination of the objective loss functions of the above tasks:

[0132]

[0133] in, This represents the final pre-training loss function. The loss function derived from steps 4.1.1, 4.1.2, and 4.1.3;

[0134] 4.2 For on-demand fine-tuning, given a pre-trained IE-Triple model, fine-tuning is used to allow the model to quickly adapt to triple extraction for specific scenarios. Given a labeled corpus, the cross-entropy loss function is used to fine-tune the IE-Triple model. The loss function for the fine-tuning stage is defined as follows:

[0135]

[0136] in, This represents the loss function during the fine-tuning phase.<t′,s′,y′> Let y' represent the sample to be tested, t' represent the original text to be tested, s' represent the guided triplet structure to be tested, and D represent the original text to be tested. task Let represent a corpus for a specific scenario, D represent the complete set of samples in the pre-training process, and p(·) and q(·) are both conditional probability distributions. and To adjust the parameters.

[0137] To verify the effectiveness of the IE-Triple model, this invention will conduct experiments and evaluations on the model, with the specific process as follows:

[0138] Step 1: Given the experimental dataset:

[0139] To evaluate the IE-Triple model, this invention uses four publicly available information extraction task datasets for experiments. The datasets used include NYT, DUIE-CHS, WebNLG, and ACE05. The NYT dataset is a relation classification dataset, with its natural language text derived from the New York Times' annotated corpus. The DUIE-CHS dataset is a large-scale, manually annotated dataset containing over 210,000 Chinese sentences extracted from Baidu Encyclopedia and Baidu News summaries, containing over 400,000 triples and 50 relation types. The WebNLG dataset comes from the Natural Language Generation Community; each dataset consists of triples describing facts and the corresponding natural language text. The ACE05 dataset, released by the Language Data Consortium, is a dataset composed of entity, relation, and event annotations, serving triple extraction tasks for multiple languages ​​including Chinese, English, and Spanish. For NYT, this dataset only labels entities in the natural language text and does not extract the corresponding triples from the text. Therefore, this invention employs a named entity recognition tool to identify entities within sentences, performs string matching between the identified entities and entity names in Freebase (Wikipedia Open Knowledge Graph) triples, and pairs sentences containing both the triples and their beginning and end entities. Detailed statistical data for the four experimental datasets are as follows: Figure 4 As shown.

[0140] Step 2: Specify experimental evaluation metrics:

[0141] 2.1 For the IE-Triple model, this invention uses information extraction task metrics to evaluate the model's performance. Precision, Recall, and F1 score are used to evaluate the experimental results. In the experiments, this invention constructs a new experimental dataset for each benchmark dataset, as shown below:

[0142] S = {[ <T,h t ,e h ,r,t t ,e t >,x]|x∈{0,1}}

[0143] Where S represents the experimental dataset, x represents whether the extracted triples are correct (0 indicates error, 1 indicates correct), T represents the original text, and h t Indicates the header entity type, e h'r' represents the head entity, 'r' represents the relation, and 't' represents the t-head entity. t Indicates the tail entity type, e t This indicates the tail entity.

[0144] 2.2 A triple is considered a correct extraction result only if the head entity, tail entity, and relation type in the triple are all correctly identified. The formulas for calculating Precision, Recall, and F1 scores are as follows:

[0145]

[0146]

[0147]

[0148] In this system, TP (True Positive) represents the number of correctly identified positive triplets; FP (False Positive) represents the number of incorrectly identified positive triplets; FN (False Negative) represents the number of incorrectly identified negative triplets; and TN (True Negative) represents the number of correctly identified negative triplets. Precision is the percentage of correctly identified triplets that are actually positive, recall is the percentage of triplets that are actually positive that are correctly identified, and the F1 score is the harmonic mean of precision and recall, serving as a combined evaluation index.

[0149] Step 3: Environment Configuration and Parameter Settings

[0150] 3.1 The method model proposed in this invention is built based on the PyTorch framework. PyTorch is a deep learning computing framework launched by Facebook AI. The hardware configuration and software environment used in the experiment are as follows: Figure 5 As shown.

[0151] 3.2 This invention uses the Adam optimizer to optimize the model parameters. The specific settings are as follows: the learning rate range is set to {0.00005, 0.0001, 0.0003, 0.0005}, and a linear scheduler with a preheating learning rate of 6% is used.

[0152] 3.3 For the encoder training of IE-Triple, the number of encoders is set to 12, and 5 to 10 negative triples are randomly generated for each natural language text.

[0153] 3.4. For the decoder training of IE-Triple, the number of decoders is set to 12.

[0154] 3.5. For the mask part of the decoder, set the masking rate to 15% and the average length of the mask to 3.

[0155] 3.6 The hidden state dimension of the IE-Triple model ranges from {2048, 2816}.

[0156] 3.7 The number of attention heads is set to 12, and the dimension of the attention heads is set to 64.

[0157] 3.8 Set the total length of the triplet pattern guidance structure p and the original text w during pre-training, as well as the length of the triplet generation structure TGS, to 128.

[0158] 3.9 The training epochs are set to 50, and the batch size for each epoch is set to 128.

[0159] Step 4: Analysis of Performance Comparison Experiment Results

[0160] 4.1 To verify the effectiveness of the IE-Triple model, this invention selected several representative end-to-end triple extraction benchmark models and conducted a comprehensive performance comparison on four experimental datasets: NYT, WebNLG, DUIE-CHS, and ACE05. The benchmark models specifically included: CopyRE, CasRel, ETL-span, GraphRel, OrderCopyRE, PRGC, RSAN, T5-base, TPLinker, and WDec. Among them, CopyRE, OrderCopyRE, and WDec are based on copying mechanisms; CasRel is based on joint decoding; ETL-span is based on a span-based labeling scheme; GraphRel is based on graph convolutional neural networks; PRGC is based on latent and global correspondences; and RSAN, TPLinker, and T5-base are based on attention mechanisms. Figure 6 The bold text indicates the best results in the experiment, the underlined text indicates the second-best results, the hyphen (-) indicates that no data was provided for the baseline model, and the text modified with "base" indicates a pre-trained model. Pre. and Rec. are abbreviations for precision and recall, respectively.

[0161] 4.2 According to Figure 6 The performance comparison experiment results are analyzed as follows:

[0162] 4.2.1 The IE-Triple model (IE-Triple-base in the figure) outperforms other benchmark methods in almost all evaluation metrics. Although the IE-Triple model is slightly lower than other best models in some precision or recall metrics, it shows the best performance in the F1 evaluation metric. Therefore, the overall performance of the IE-Triple model is superior to other benchmark models. In particular, the F1 score of the IE-Triple model is generally 0.7% to 2% higher than the second-best model on different datasets, which indicates that the model proposed in this invention has strong robustness.

[0163] 4.2.2 Although the TPLinker model is more complex than the CasRel model, it only achieved a 0.1% improvement on the WebNLG dataset. The model proposed in this invention shows a significant improvement over TPLinker on this dataset. This invention argues that the triple-guided structure of the IE-Triple model includes information on both entity type and relation type, effectively reducing redundant relations in the dataset, especially in the WebNLG dataset which contains hundreds of relation types. Compared to models that extract triples solely based on relation information, the IE-Triple model eliminates redundant relations based on entity type, thereby improving model performance.

[0164] 4.2.3 Large-scale model pre-training provides the foundation for the IE-Triple model to handle triple extraction in various scenarios. Pre-training the IE-Triple model on a large dataset allows the model to share semantic knowledge from different datasets. This is especially true for the DUIE-CHS dataset, where knowledge is rarely found in the pre-training dataset. This further demonstrates that the IE-Triple model can perform triple extraction across domains, i.e., it has the ability to share learning across various triple extraction tasks.

[0165] Step 5: Verify the effectiveness of fine-tuning

[0166] 5.1 To verify the ability of the IE-Triple model to quickly adapt to various scenarios after fine-tuning, this invention uses four original datasets as the datasets for downstream tasks and conducts Few-Shot experiments. Few-Shot experiments are a type of low-sampling experiment, where classification / regression is performed with very few samples. Specifically, this invention samples 1 / 4 / 8 samples from the data pairs in the training set and uses the F1 score as the result of each experiment. For the selected models, those modified with "base" indicate the initial model, and those modified with "Fine-tuned" indicate the fine-tuned model. Furthermore, to avoid the influence of random sampling, this invention repeatedly samples different samples from each dataset and conducts multiple experiments, recording the average results.

[0167] 5.2, according to Figure 7 The results of the fine-tuning effect comparison are analyzed as follows:

[0168] 5.2.1 In addition to comparing the initial IE-Triple model (IE-Triple-base) and the fine-tuned IE-Triple model, the T5 model was used as the control group in the experiment. The fine-tuned T5 model showed significant performance improvement in various data experiments, which confirms that the fine-tuning strategy can enable the model to quickly adapt to data in various scenarios and has universality applicable to different models.

[0169] 5.2.2. Regarding the comparison between the IE-Triple and T5 models, the IE-Triple model improved by an average of 15% to 17% on various datasets compared to the T5 model. This is attributed to the pattern hint mechanism of the triple pattern guidance structure in the IE-Triple model.

[0170] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A knowledge graph construction method based on IE-Triple, characterized in that, The method includes the following steps: Step 1: Given the original text and the triplet pattern guide structure; Step 2: The triplet pattern structure guide embeds a triplet pattern guide structure into the original text and uses it as input to the IE-Triple model, which consists of a triplet pattern structure guide and a triplet pattern structure generator. Step 3: The triplet pattern structure generator extracts information from the original text in the input and generates triplets with corresponding pattern structures based on the triplet guided structure transformation. Step 4: Provide IE-Triple model pre-training and fine-tuning strategies to enable the model to adapt to downstream tasks; The process of step 1 is as follows: 1.1 Given the original text ,in, For the first Word vectors; 1.2 Given a triplet schema bootstrapping structure (TIS) to indicate the entity type and relation type associated with the target information, defined as follows: ； in, This is a triplet schema guiding structure. The markers [head], [Relation], and [tail] represent the location information of the head entity, relation, and tail entity in TIS, and the markers EntityType and RelationType specify the type of the corresponding entity and relation. The process of step 2 is as follows: For the original text The Triple-schema Structural Instructor (TSI) embeds TIS as a prefix into... In the formula, the input to the IE-Triple model is given as follows: ； in, This represents the value calculated by TSI. [SEP] represents the vector concatenation operation. Separation information from TIS; Among them, TSI is a pattern structure-based prompting mechanism that can guide the IE-Triple model to generate corresponding guided triples. This mechanism enables the model to adaptively control and recognize head entities, tail entities and the relationship between them, as well as share semantic knowledge in different labels and tasks. The process of step 3 is as follows: 3.1 In order for the IE-Triple model to generate valid triples, a triple pattern generation structure TGS needs to be defined. TGS consists of two parts: 1) Entity Part: Predefined entities identified from the original text, including head entity and tail entity; 2) Relation Part: Predefined relationships identified from the original text, including the relationship between head and tail entities and the relationship between entities and attributes. Each TGS contains three types of tokens: 1) EntityType*: indicating the specified type of the head and tail entities in the triple; 2) HeadName / TailName: indicating the actual value of the head or tail entity extracted from the original text; 3) RelationType*: indicating the specified type of the relation in the triple; In addition, the ":" in TGS indicates the mapping from the specified type of the entity to the actual value, and the two structure indicators "{" and "}" are used to encompass all the information of a single triple; 3.

2. The value calculated by TSI is used as the model input, and the target triples are extracted by TSG.

2. The knowledge graph construction method based on IE-Triple as described in claim 1, characterized in that, The process described in section 3.2 is as follows: 3.2.1 The encoder first performs positional encoding on the input vector, marking each position of the input vector. This involves superimposing the positional encoding onto the original input vector, resulting in an encoded vector with the same dimension as the input vector. The formula is as follows: ； in, Represents the encoded vector. Indicates positional encoding; 3.2.2 The encoder consists of multiple independent attention heads. These attention heads are implemented based on the Scaled Dot-Product Attention mechanism, an adaptive weighted attention mechanism. This mechanism is described as a mapping from the input vector to the hidden state through query vectors, key vectors, and value vectors. The query vector, key vector, and value vector are generated as a linear combination of the input vectors, as shown in the following formula: ； ； ； in, For query vector, For key vectors, For value vectors, , , This is the weight matrix; 3.2.3 The vector output by the attention mechanism is a weighted combination of value vectors. This weight matrix is ​​generated by comparing the query vector and the key vector, reflecting the proportion of words in the original text that are related to the triple information. Furthermore, the attention head is divided equally according to the input dimension. (The division is denoted as...) portion for , put the first The hidden state generated by each attention point is denoted as The formula is as follows: ； in, Represents the normalized exponential function, Represents the input vector Dimensions Indicates matrix transpose; 3.2.

4. The output of each attention head in step 3.2.3 The process involves splicing, followed by residual connections and hidden layer regularization. The residual connections are part of step 3.2.

1. and The linear superposition of the hidden layers, the hidden layer regularization normalizes the hidden layers in the network to a standard normal distribution, and then the output is passed through a bit-by-bit feedforward network to finally obtain the encoder hidden state, as shown in the following formula: ； in, Indicates the encoder is hidden. This represents a feedforward neural network. This represents the hidden layer regularization function; 3.2.

5. For the decoder, the position-encoded vector is subjected to masked multi-head attention computation. The masking operation is to preserve the autoregressive property, that is, to ensure that the prediction result depends only on the generated output words. The masked multi-head attention mechanism, residual connections, and hidden layer regularization are regarded as the front-end of the decoder, denoted as... The calculation formula is as follows: ； in, Indicates the first i Each word segmentation vector This indicates the first iteration of the loop. This operation. This represents the hidden layer regularization function. Will The vectors are masked and then processed using a multi-head attention mechanism. express The vector generated by position encoding. Represents the query vector, key vector, and value vector that participate in the mask multi-head attention computation; 3.2.6 The back-end structure of the decoder is the same as that of the encoder, sequentially passing through a multi-head attention mechanism, residual connections and hidden layer regularization, a bit-by-bit feedforward network, and residual connections and hidden layer regularization. The input to the back-end part comes from the hidden state of the encoder in step 3.2.4 and the vector generated by the front-end part of the decoder in step 3.2.

5. The back-end part of the decoder is denoted as... Finally, the vectors output by the backend are fully connected and summed. Function processing yields the predicted probability. The calculation formula is as follows: ； in, express With TGS The matching probability of each token. Based on Fully connected layer of functions This indicates the hidden state from the encoder in step 3.2.

4. This represents the result calculated in step 3.2.

5. Finally, the content of the TGS expression generated by the model prediction is output in text form.

3. The knowledge graph construction method based on IE-Triple as described in claim 1, characterized in that, In step 4, firstly, relevant large datasets are collected from the Internet and these data samples are preprocessed, including structured data, semi-structured databases and unstructured data. These samples are processed into text-to-TSG (text-to-TSG) structures and used as pre-training samples for the model. Structured data is transformed according to its relational patterns, while unstructured data requires manual transformation. To ensure the validity of unstructured data, public knowledge bases provided by Wikipedia and Baidu Baike are used. Then, the IE-Triple model is pre-trained uniformly using the above dataset. Finally, the pre-trained IE-Triple model is adapted to downstream tasks according to the needs of real-world scenarios. Since the pre-trained IE-Triple model is based on a general knowledge base, it has the ability to share knowledge in triple extraction tasks in different scenarios.

4. The knowledge graph construction method based on IE-Triple as described in claim 3, characterized in that, The process of step 4 is as follows: 4.1 The pre-training of the IE-Triple model is divided into three parts: 1) pre-training of the encoder part of the IE-Triple model; 2) pre-training of the decoder part of the IE-Triple model; 3) pre-training of the IE-Triple model as a whole. For this purpose, a loss function is defined for each of the above three pre-training tasks. 4.2 For on-demand fine-tuning, given a pre-trained IE-Triple model, fine-tuning is used to allow the model to quickly adapt to triple extraction for specific scenarios. Given a labeled corpus, the cross-entropy loss function is used to fine-tune the IE-Triple model. The loss function for the fine-tuning stage is defined as follows: ； in, This represents the loss function during the fine-tuning phase. Indicates the sample to be tested. Indicates the prediction result. This represents the original text to be tested. This indicates the guiding structure of the triplet to be tested. Corpora representing specific scenarios, This represents the complete set of samples used in pre-training. and All are conditional probability distributions. and To adjust the parameters.

5. The knowledge graph construction method based on IE-Triple as described in claim 4, characterized in that, The process of step 4.1 is as follows: 4.1.1 For the pre-training of the IE-Triple encoder, in order to enable the encoder part to capture the mapping ability from the original text to the hidden layer state, that is, whether there is information related to the triple guidance structure in the original text, the data involved is the text data of the research object. and triplet guidance structure Treat these two types of data as data pairs During model training, if only positive samples are provided to the model, the model will only remember the correct triples from the pre-training data. Therefore, negative samples need to be constructed through random matching. The target loss function for the IE-Triple encoder pre-training is given below: ； in, This represents the target loss function for encoder pre-training. This represents the output of the model during this pre-training process. These are the positive samples used for pre-training the encoder. For the negative examples used in the encoder pre-training, It is a logarithmic function. For conditional probability distribution, These are the parameters of the encoder; 4.1.2 For the IE-Triple decoder pre-training, the decoder's role is to parse the hidden states generated by the encoder into semantic representations of target triples. By modifying the triple pattern generation structure to generate negative examples, the decoder's ability to effectively recognize positive examples is pre-trained. Therefore, the target loss function for IE-Triple decoder pre-training is given: ； in, This represents the target loss function for the decoder pre-training. Indicates the data to be tested. express The One token, This indicates that the length of the data to be tested is... substring, These are positive samples used for pre-training the decoder. The negative samples are used for pre-training the decoder. It is a logarithmic function. For conditional probability distribution, These are the parameters of the encoder; 4.1.3 For the overall pre-training of the IE-Triple model, in order to effectively train the extraction capability of the entire model, training samples are constructed in the form of text-to-target triplet structures. A portion of the words in the original text is masked to generate the masked portion. Since the input of the model already contains the output content, it is necessary to rearrange the word order of the marked words in the original text and the word order of the triplet pattern generation structure markers. The overall pre-training target loss function for the IE-Triple model is given below. ； in, This represents the target loss function for the overall pre-training. This represents the structure for generating triplet patterns. Represents the original text. Generate a structure for the triplet pattern after the word order is rearranged. Let D be the original text after word order rearrangement, and let D be the full sample set. It is a conditional probability distribution; 4.1.4 Based on the objectives of the three pre-training tasks above, the final objective loss function is a combination of the three pre-training objective loss functions: ； in, This represents the final pre-training loss function.

Images