A method for extracting complex relations from a small number of samples based on adaptive intervals of data distribution
By introducing adaptive intervals and negative sample mechanisms of data distribution in few-sample learning, the prediction problem of multiple relational triplets in complex sentences is solved, and the prediction ability and confidence of the model in complex scenarios is improved, especially In the case of handling nested relationships and data imbalance.
Patent Information
- Application Number
- CN202111422588.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-26
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-11-26
AI Technical Summary
The existing few-sample learning method works well when dealing with simple sentences, but does not perform well in complex sentences, especially when multiple relational triplets are included, the actual scenario is not fully considered, and there are single entity overlaps and Nested relationship prediction ability is insufficient.
Using a method of extracting complex relationships based on data distribution adaptive intervals, the extraction of complex relationships can be achieved by constructing metatasks, metric learning and negative sample mechanisms, and multiple entities and multiple relationship triplets in text can be processed. and adjust the measurement spatial boundaries through data distribution adaptive intervals to improve the prediction confidence of the model.
This method can effectively deal with the prediction of multiple relational triplets in complex sentences, solve the prediction problems of overlapping and nested relationships of a single entity, improve the confidence of the model's prediction results in the case of data imbalance, and enhance the None-of -the-above scenario prediction capability.
Smart Images

Figure CN114138935B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer natural language processing and is a method for extracting complex relations from a small number of samples by adjusting the metric space division range based on the internal data distribution of a task. Background Art
[0002] With the rapid development and popularization of computers and the Internet, in this era of information explosion, how to quickly analyze and process information and extract valuable information from text has become a research hotspot and a problem that needs to be solved urgently. Among them, information extraction technology converts natural language text described in unstructured or semi-structured descriptions into structured data, and plays an important role in many application scenarios such as domain knowledge graph construction, task-based dialogue systems, and knowledge question-answering systems. It is of great significance to extract valuable information from a large amount of data through information extraction so that it can be understood by machines.
[0003] Information extraction is an application of natural language processing, and its task is to automatically extract structured information from unstructured or semi-structured documents. By modeling document information, computers can extract important information from a large amount of text, saving a lot of manpower and material resources.
[0004] Relation extraction is an important task in the field of information extraction. Its purpose is to extract the semantic relationship between specified entity pairs from natural language text. For example, there is a relationship "country" between "Chinese Navy" and "China". Relation extraction has a wide range of application scenarios in subsequent applications such as automatic construction of knowledge graphs, automatic question answering, and text mining.
[0005] Relation extraction methods based on deep learning can be roughly divided into supervised learning and unsupervised learning. Supervised learning often relies on a large amount of high-quality manually annotated data. In reality, a large amount of annotated data consumes a lot of human resources, and professional content is also difficult to obtain. Therefore, some people have proposed unsupervised learning methods, but the accuracy of this type still needs to be improved. Therefore, it is of practical significance to study the relationship extraction method under data constraints.
[0006] There are two solutions to this problem. One is to increase data. Specific methods include remote supervised learning, semi-supervised learning, active learning, and text data enhancement. The idea of remote supervision is to use a large semantic database to automatically obtain relationship type labels and align the triples in the existing knowledge base to the rich unstructured data. This method is based on a very strong assumption that for a triple in an existing knowledge base, it is assumed that any sentence in an external document containing this pair of entities reflects this relationship to a certain extent. This assumption is too absolute and will inevitably introduce noise.
[0007] Another method is to improve the utilization of limited data, that is, the few-shot learning method. In few-shot learning, unlike deep learning, training is performed based on a large number of meta-tasks to obtain a good initialization parameter. Based on this, only a small amount of data is needed to achieve good results for the categories in the new meta-task. The meta-tasks are divided into meta-tasks in the training phase and meta-tasks in the test phase, which are usually set to N-way K-shot. N represents the number of categories in the learning task, and K represents the number of samples in the support set. The meta-task contains a support set and a query set. The model is trained on the support set and then tested on the test set.
[0008] Currently, the relation extraction method using few-shot learning works well in processing simple sentences, but performs poorly in complex sentences containing multiple relation triplets, and does not fully consider the situation of complex sentences in actual scenarios. Summary of the invention
[0009] The purpose of the present invention is to provide a few-sample complex relationship extraction method based on data distribution adaptive interval to address the limitations of existing few-sample learning. The method realizes few-sample relationship extraction based on metric learning and can be used in situations where there are multiple entities in the text, the entities may be nested entities, and there may be multiple relationship triplets. It can also increase the prediction probability of categories with more data distribution and reduce the prediction probability of categories with less data distribution.
[0010] In order to achieve the above object, the technical solution adopted by the present invention is as follows:
[0011] A method for extracting complex relations from a small number of samples based on adaptive intervals of data distribution is proposed. By constructing meta-tasks, complex relations are extracted based on adaptive intervals of data distribution and negative sample mechanisms.
[0012] Further, the following steps are included:
[0013] Step 1: Divide the training data containing text instances and relation triplets into a series of meta-tasks in an N-way k-shot manner and store them in the task list [Task0, Task1, …]. Each meta-task contains a support set and a query set, and the ratio of the number of instances in the support set to the number of instances in the query set is 2:1 to 3:1.
[0014] Step 2: construct a pre-trained network for the encoding phase, process the data in each meta-task, and obtain the encoding of the text;
[0015] Step 3: Encode the text of the support set / query set obtained in step 1 to obtain a vector representation; take the vectors of the corresponding positions of the entities in the current instance, aggregate them, and obtain the vector representation of each entity;
[0016] Step 4: Combine and concatenate the entity vector representations in the support set / query set in pairs to obtain [e i ,e j ], i and j are entity serial numbers, i≠j;
[0017] Step 5: The support set / query set obtained in step 1 contains N+ relation categories [R1, R2, …, R N+ ], the entity pair representations obtained in step 4 are stored according to the corresponding relationship. If there is no relationship between the entity pairs, they are stored as the None category, and the mean of the entity pair representations under each type of relationship is obtained as the vector cluster center of each type of relationship, that is, the vector representation of each type of relationship, thereby obtaining the metric space;
[0018] Step 6, calculating the Euclidean distance between the representations of all entity pairs to be predicted in the support set and the query set obtained in step 4 and the center of each type of relationship cluster in the metric space, and performing relationship category prediction through an activation function;
[0019] Step 7: Run a large number of meta-tasks, repeat steps 2 to 6, train the pre-trained network in the encoding phase by gradient descent, and update the network parameters.
[0020] Compared with the prior art, the advantages of the present invention are as follows:
[0021] 1. The present invention solves the limitation that the existing methods of few-sample relation extraction can only extract single sentences and single relations. By making multiple predictions by combining entity pairs with concatenated vectors, the prediction of multiple relation triplets contained in an instance can be solved.
[0022] 2. Compared with the existing few-sample relationship extraction methods, the present invention can also solve the situation where there is a single entity overlap between triples.
[0023] 3. There may be nested situations between entities, such as there is a nested relationship between the two entities "Chinese Navy" and "China". At this time, if there is a relationship triple <Chinese Navy, affiliated country, China>, the present invention can still have predictive capabilities for this scenario.
[0024] 4. Most of the existing few-sample methods meet the N-way K-shot setting, which is easy to meet in the case of a single sentence and a single relationship, but cannot achieve data balance in complex scenarios, that is, when an instance contains multiple relationships. The present invention introduces an adaptive s-interval based on the data distribution in the meta-task, which alleviates the imbalance of data distribution and improves the confidence of the model prediction results.
[0025] 5. The present invention introduces a negative sample mechanism to separately partition the metric space of the set for entities that do not have any relationship, thereby improving the prediction ability of the model for the None-of-the-above scenario. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 It is a structural diagram of the overall framework of relationship extraction in an embodiment of the present invention.
[0027] Figure 2 It is a flow chart of extracting complex relations from a small number of samples based on adaptive intervals of data distribution in the training phase in an embodiment of the present invention;
[0028] Figure 3 is a schematic diagram of an encoder in an embodiment of the present invention;
[0029] Figure 4 is a schematic diagram of converting an entity vector into a fixed dimension in an aggregator step in an embodiment of the present invention; DETAILED DESCRIPTION
[0030] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings.
[0031] A method for extracting complex relations from a few samples based on adaptive intervals of data distribution, in which the few-sample learning can be quickly generalized to new tasks that only contain a small number of samples with supervised information by using prior knowledge. This method extracts complex relations from a few samples based on metric learning in the few-sample learning method.
[0032] Among them, a complex relationship is defined as a text containing multiple entities, denoted as [E1, E2, …, E e ], where e is the number of entities in the current text, and the relationship in the text can be recorded as [R1, R2, …, R r ], where r is the number of all relationship categories contained in the complete dataset. There may be multiple relationships between different entities, such as<e1,R1,e2> ,<e3,R2,e5> ,<e2,R3,e3> etc.; relation triplets may not interfere with each other, such as the triplets<e1,R1,e2> ,<e3,R2,e5> ; there may also be overlaps, such as<e1,R1,e2> ,<e2,R3,e3> , where the e2 entity is the overlapping entity.
[0033] The overall model structure of the metric learning method is as follows Figure 1 For a given instance (i.e., text) and entities in the instance, a meta-task is first constructed. Each meta-task contains a support set and a query set. The support set is used to construct a metric space, such as Figure 1As shown in the lower part, the text vector is obtained through the encoder, and then the vector representation of each entity is obtained through the aggregator. The entity vectors are concatenated two by two and classified into the belonging relationship, and the cluster center vector under each type of relationship is obtained, which represents the vector representation of each type of relationship in the metric space; and the query set is used to calculate the distance between the current entity pair to be predicted and each cluster center in the metric space, such as Figure 1 As shown in the upper part, the text also passes through the encoder and aggregator to obtain the vector representation of each entity, which is concatenated two by two to obtain the vector representation of the entity pair to be predicted, and the Euclidean distance is calculated with the relationship vector in the metric space (i.e., each cluster center vector); the vector similarity is judged according to the distance, and the relationship category corresponding to the closest cluster center is selected to obtain the prediction result; and the loss function is calculated to optimize the metric space mapping function. The process of metric learning is the process of obtaining the optimal metric space mapping function through several learning tasks.
[0034] The specific steps of this method are as follows:
[0035] Step 1) First, divide the data into a series of tasks according to the idea of N-way k-shot, that is, first randomly sample N relations in the relation category of the data set, and then randomly sample K data in the data containing these relations. Since the scenario of the present invention is a complex relationship, that is, a piece of data contains multiple relations, the meta-task obtained by sampling may introduce new relations outside the sampled N relation categories. The number of relations in the current task is greater than or equal to N, which is recorded as N+relationship categories. In this way, a meta-task is obtained, which is repeated many times to obtain multiple meta-tasks. Stored in the task list [Task0, Task1,…, Task n ]. Each task contains a support set and a query set, and the ratio of the number of instances in the support set to the number of instances in the query set is 2:1 to 3:1.
[0036] Step 2) Construct a pre-trained network for the encoding phase to obtain the encoding of the text.
[0037] Step 3) Encode the text of the support set obtained in step 1) to obtain the vector representation of each text instance [x1, x2, …, x n ], where n is the length of the text, and each character of the text corresponds to a vector. Take the vectors corresponding to the starting position to the ending position of the entity in the current instance, aggregate them, and convert them into a fixed dimension to obtain the vector representation of each entity. The vector representation of the i-th entity is recorded as e i .
[0038] Step 4) Combine and concatenate the entity vector representations of each text in the support set in pairs to obtain [e i ,e j ], the value range of i and j depends on the number of entities e in the current text, 1<=i, j<=e.
[0039] Step 5) The support set obtained in step 1) contains N+ relation categories [R1, R2, …, R N+ ], the entity pair representations obtained in step 4) are stored according to the corresponding relationship. If there is no relationship between the entity pairs, they are stored in the None category, and the mean is calculated as the vector cluster center of each type of relationship, that is, the vector representation of each type of relationship. Thus, the metric space is obtained.
[0040] Step 6) The query set obtained in step 1) is processed from step 2) to step 4) to obtain the representations of all entity pairs to be predicted, and the Euclidean distance is calculated with the center of each type of relationship cluster in the metric space to make predictions.
[0041] Step 7) Run a large number of meta-tasks, repeat steps 2) to 6), calculate the cross entropy loss function, train the network by gradient descent, and update the parameters.
[0042] Among them, the method of constructing meta-task in step 1) needs to be distinguished from the existing technical solutions.
[0043] 1-1) First, the data is processed and stored according to the relations contained in each instance. Each sentence instance contains the natural language original text of the sentence, the named entities appearing in the sentence, and the relation triples appearing in the sentence.
[0044] 1-2) Divide the relationship categories in the data into 8:2 ratios, where the first 80% of the relationship categories are taken as training data and the last 20% are taken as test data.
[0045] 1-3) According to N way-K shot sampling, N relation categories are sampled in the training data, and K data are sampled in each category as the support set. Because each data contains multiple relations, this method will inevitably introduce new relation categories. Therefore, it is necessary to re-count the relation categories included in the current meta-task, as described above, denoted as [R1, R2, …, R N+ ].
[0046] 1-4) The ratio of the number of instances in the support set to the number of instances in the query set is 2:1 to 3:1. According to this ratio, K / 2 pieces of data are sampled as the query set under the same N categories, and it is ensured that there is no intersection between the data in the query set and the data in the training set.
[0047] Step 2) Use the training phase meta-task constructed in step 1) to train the pre-trained language model in the encoding phase:
[0048] For natural language sentences, each character is first converted into an index representation in the dictionary and then encoded. A pre-trained model is used. In this encoding model, a d-dimensional vector is generated for each character of each sentence by inputting a sentence sequence.
[0049] Step 3) Obtain the entity representation according to the text vector representation obtained in step 2) and aggregate it to a fixed dimension:
[0050] In this framework, since most named entities are composed of multiple characters, after the encoder converts each character into a vector, it is necessary to aggregate these character vector representations so that the representation of the named entity can be represented by a vector of fixed dimension, which is convenient for subsequent processing. In order to take into account the information of each character in the entity, average pooling combined with activation function row aggregation is used, and the dimension after aggregation is still d-dimensional. i The vector representation of is:
[0051] e i =LeakyRelu(Avg(Encoder(x start , …, x end )))
[0052] Among them, encoder refers to the pre-trained model used in step 2), (x start , …, x end ) refers to the current entity e i The starting and ending positions in the example are recorded as start and end respectively. The activation function can be flexibly selected, and LeakyReLu is temporarily used here.
[0053] Step 4) In the meta-task of the training phase, the entity representations of each training instance contained in the support set are concatenated in pairs:
[0054] The entity representations are concatenated in pairs to obtain multiple pairs of vector representations of dx 2 dimensions. i ,e j ), where e is the embedding of the entity, and the relationship between two entities is represented by the concatenation of their vectors, that is,
[0055]
[0056] Step 5) According to the entity pair representations obtained in step 4), the mean is calculated according to the corresponding relationship, and the mean is used as the vector cluster center of each type of relationship, that is, the vector representation of each type of relationship. Thus, the metric space is obtained.
[0057] 5-1) For each relationship category, combined with the entity pair representations contained in the category, the cluster center of each relationship category can be generated as the vector representation of the relationship. Vector representation of relationship k The calculation method is as follows,
[0058]
[0059] Among them, S kr is the concatenated set of entity pair vectors that have relation k.
[0060] 5-2) Thus, the preliminary division of the metric space is obtained. In this step, this solution designs an adaptive interval based on data distribution to alleviate the data imbalance problem in the meta-task. Due to the particularity of complex sentences, there are more instances of some relations in the support set. Considering this imbalance, a division boundary is introduced. This boundary is used in relation prediction to increase the probability of predicting the correct relation label. For relation k, its boundary threshold is defined as follows:
[0061]
[0062] Among them, |S k | and |S R\k | are the number of instance triples of relation k in the support set and the number of instance triples other than relation k, respectively, and R is the relation set of the support set.
[0063] 5-3) Set up a negative sample mechanism to perform calculations from 5-1) to 5-2) on entity pairs that do not have any relationship, and make predictions and judgments as independent spaces.
[0064] Step 6) The query set obtained in step 1) is processed according to step 2) to step 4) to obtain the representation of all entity pairs to be predicted, and the Euclidean distance is calculated with the center of each type of relationship cluster in the metric space, and prediction is performed. The loss function is calculated to update the parameters:
[0065] 6-1) Calculation of likelihood function:
[0066] Get the entity pair concatenation r to be predicted in the query set i,j , let it be instance x, then the likelihood of it having relation y is calculated as follows:
[0067]
[0068] Where S is the support set, d is the Euclidean distance, m is the boundary threshold based on data distribution, and R is the relationship set in the task.
[0069] 6-2) Calculation of loss function:
[0070] By calculating the likelihood function in formula 5-1), we can obtain the predictions of entities i and j in the sentence for each relationship in R, and use the cross entropy function to calculate the loss:
[0071]
[0072] r′ i,j,k =p(y=k|x,S)
[0073]
[0074] where r′ i,j,k is the probability that there is a relationship k between entity i and entity j, r i,j,k is the true label of the relationship k between entity i and entity j, and Q is the set of all entity pairs in the sentences of the query set.
[0075] The multiple meta-tasks constructed in step 1) are respectively executed in steps 2) to 6), and the network is trained by gradient descent to update the parameters.
[0076] Among them, the meta-task construction method is different from the traditional few-shot learning meta-task construction. The existing few-shot relation extraction is mostly based on a single sentence and a single relation. The meta-task construction follows the N-way K-shot experimental setting, that is, N categories are sampled from the relation category, and K data are sampled from each category. Based on complex relation scenarios, a sentence instance contains multiple relation triplets, and it is difficult to ensure that the task construction follows N-way K-shot. Step 1 follows the existing idea and will inevitably introduce new relations outside the N categories. The scheme counts the new relations into the relation set, recorded as [R1, R2, …, R N+ ], and newly introduced relations are also predicted in the query set stage.
[0077] The encoder model is diverse. You can use Bert or other pre-trained models such as Albert, Spanbert, etc., or you can use word vectors such as word2vec and GloVe for encoding.
[0078] Obtain a fixed-dimensional vector representation of the entity, obtain the start and end positions of the entity in the given instance according to the entity contained in the given instance, take the start and end position vectors corresponding to the sentence vector obtained in step 2, that is, the vectors of multiple characters in the entity in step 3. Step 3 processes the vectors of multiple characters into fixed dimensions. Nested entities can also be represented separately by the above method.
[0079] The method for obtaining the relational representation in step 4 combines all entities in pairs and concatenates the corresponding vectors in order. This method is based on the entity pair representation as the prediction unit, and can predict the triples with reversed entity pairs, and can predict the complex situation where an instance contains multiple relation triplets.
[0080] In step 5, a special mechanism is introduced for negative sample entity pairs. In the metric space, in addition to the metric space of N+ categories included in the meta-task, a separate negative sample space is divided. This method can solve the problem of predicting relation triples in the None-of-the-above scenario, that is, when there is no relation.
[0081] Step 6 introduces an adaptive interval based on data distribution, i.e., a boundary threshold, in the prediction stage. Step 6 adjusts the boundary of the metric space division to address the problem of data imbalance in the meta-task in complex relationship scenarios, expands the boundary of the metric space corresponding to the relationship with more data distribution, and conversely, reduces the metric space with less data distribution. Give the confidence level of each type of relationship prediction to improve the model effect.
[0082] There are many ways to obtain a vector representation of a fixed dimension of an entity. You can use average pooling, maximum pooling, etc., and introduce activation functions to increase nonlinear features. The activation function can also be diverse, such as using LeakyReLu.
[0083] In summary, in the training phase, this method obtains the vector representation of the entity through the pre-trained model encoder and the aggregator that converts the entity into a fixed dimension. The entity representations of the support set in the meta-task are concatenated pairwise to obtain the cluster center as the relationship representation, and the entity representations of the query set in the meta-task are concatenated pairwise as the entity pairs to be predicted. The prediction probability of each type of relationship is measured by calculating the Euclidean distance with the relationship representation of the support set in the metric space, and the loss function is calculated to update the parameters; the test phase is similar to the training phase. The meta-task in the test phase, the support set is used to construct the metric space of the current task, and the query set is used to make predictions in combination with the metric space. The difference from the training phase is that the query set directly outputs the prediction results, and no loss function calculation and parameter update of each module are performed. Among them, the few-sample relationship extraction model based on data distribution adaptive interval can extract relationships for texts containing multiple entities and multiple relationship triplets in few-sample scenarios.
[0084] Aiming at the complex scenarios where the text contains multiple entities and multiple relations, this method designs a meta-task construction method, introduces adaptive intervals based on data distribution, and uses this to adjust the boundary of the metric space. At the same time, it introduces a negative sample mechanism to solve the problem of predicting relation triplets where there is no relation in the None-of-the-above scenario, and can also perform relation extraction for common entity nesting situations.
[0085] Here is a more specific example:
[0086] A method for extracting complex relations from a small number of samples based on adaptive intervals of data distribution. The overall process of the training phase is as follows: Figure 2 As shown, it includes the following steps:
[0087] Step 1) The dataset is divided by task. Each sentence instance in the dataset contains the natural language original text of the sentence, the named entities appearing in the sentence, and the relation triples appearing in the sentence. The main steps include:
[0088] 1-1) Read the data set and use Python to process the data according to the data set relationship category [R1, R2, …, R r ], the dataset containing the corresponding relationship is stored in the form of a dictionary. First, we use the relationship classification as the key of the dictionary, and its corresponding value is the sentence containing the relationship classification. Each sentence instance is stored in the form of a dictionary, and its content includes the natural language original text of the sentence, the named entities appearing in the sentence, and the relationship triples appearing in the sentence.
[0089] 1-2) The model is trained in meta-task units, which are divided into meta-tasks in the training phase and meta-tasks in the testing phase. The ratio of the training phase to the testing phase is 8:2. Therefore, the first 80% of the relationships in the relationship category number are used as data in the training phase, and the last 20% of the relationships are used as data in the testing phase.
[0090] 1-3) Each meta-task contains a support set and a query set. The meta-task construction method in the training phase is as follows:
[0091] The construction of the support set follows the idea of N-way K-shot. Since a sentence may contain multiple relations, it is difficult to meet the requirements of N-way K-shot in few-shot learning in terms of relation triples. Therefore, the dataset is concentrated, N relations are sampled, and K sentence instances containing corresponding relations are sampled respectively. The requirements of N-way K-shot are met at the sentence level. In practice, it is possible to introduce new relations outside the N relation categories in this process. Therefore, it is necessary to re-count the relation categories included in the current meta-task, and the new relations will also be counted in the support set, and the N relations are expanded to N+ (N+ is greater than or equal to N).
[0092] The construction idea of the query set is consistent with that of the support set. The instances under the relationship categories and corresponding relationships are sampled respectively, and the number of instances is reduced to K / 3~K / 2.
[0093] 1-4) The idea of constructing meta-tasks in the test phase is consistent with 1-3). The difference is that the data comes from the data in the test phase.
[0094] Step 2) Use the training phase meta-task constructed in step 1) to train the pre-trained language model in the encoding phase:
[0095] 2-1) First, use BERT's tokenizer tool to tokenize the sentence, and then use the index in the word list to represent these words to form a character sequence. The vector representation corresponding to the sentence is obtained through the pre-trained model Bert.
[0096] 2-2) Since the lengths of the character sequences of sentences are different, these sentences need to be padded before encoding. Figure 3 As shown in the figure, the length of the longest sentence in the current meta-task is taken as max_len, and the lengths of the remaining sentences need to be aligned with max_len, so 0 is added at the end to achieve padding. Since the padding operation is used, useless data such as 0 is introduced, so a mask is needed to filter out useful data. For each sentence after padding, the value of the position corresponding to its non-zero part in the mask is 1. After encoding, the vector representation of each text instance is obtained [x1,x2,…,x n ], where n is the length of the text, and there is one vector for each character of the text.
[0097] 2-3) In the dataset, named entities are known. Therefore, entities are also represented by corresponding entity masks. Labels are the annotations of the relationship between two entities. If there are N known relationships in all sentences in a task, then label is a one-dimensional vector of length N. In this vector, if two entities have relationship i and j, the element value corresponding to index i and j is 1.
[0098] Step 3) Encode the text of the support set obtained in step 1) using the method in step 2), take the vectors corresponding to the start and end positions of the entities in the current instance and aggregate them to obtain the vector representation of each entity. The main steps are as follows:
[0099] Since named entities and their contexts are mostly composed of multiple characters, after each character is converted into a vector by the encoder Bert, the characters of the entity need to be aggregated so that the vector representations of named entities of different lengths can maintain a fixed dimension. Figure 4 As shown, step 2) obtains the vector representation of the sentence, takes the vector corresponding to the start and end positions of the entity in the current instance, performs average pooling and activation function (LeakyReLU is selected here, which can be flexibly selected), and the obtained entity vector representation is still of fixed dimension, that is, 768 dimensions obtained by Bert, which is convenient for further processing. It is necessary to construct an entity mask to identify the start and end positions of the entity in the sentence in order to perform average pooling. Therefore, the position corresponding to the entity is recorded as 1 in the entity mask, and the corresponding positions of other characters are recorded as 0. Entity e i The vector representation of is as follows, where (x start , …, x end ) is the vector from the starting position to the ending position of the entity in the text instance:
[0100] e i =LeakyRelu(Avg(Encoder(x start , …, x end )))
[0101] Step 4) Combine and concatenate the entity vector representations in the support set in pairs, and obtain [e i ,e j ], the dimension is 1x(768x2), denoted as r i,j :
[0102]
[0103] Step 5) The support set obtained in step 1) contains N+ relation categories [R1, R2, …, R N+ ], for each relationship classification, combined with the entity pair representations contained in the classification, the cluster center of each relationship classification is generated as the vector representation of the relationship.
[0104] 5-1) Store the entity pair representations corresponding to each type of relationship in a dictionary. The key of the dictionary is the relationship category, and the value of the dictionary is a list. The list stores the vector representations of the entity pairs that exist in the relationship. The average of the entity pair vectors in each list is obtained to obtain the vector representation of each type of relationship. The vector representation of relationship k is The calculation method is as follows,
[0105]
[0106] Among them, S kr is the concatenated set of entity pair vectors that have relation k.
[0107] 5-2) Introduce adaptive intervals based on data distribution to alleviate the data imbalance problem in meta-tasks. Calculate the proportion of entity pairs corresponding to each type of relationship in each meta-task, and apply the dividing boundary to relationship prediction to increase the probability of predicting the correct relationship label. For relationship k, its boundary threshold is defined as follows:
[0108]
[0109] Among them, |S k | and |S R\k | are the number of instance triples of relation k in the support set and the number of instance triples other than relation k, respectively, and R is the relation set of the support set.
[0110] 5-3) Set up a negative sample mechanism to perform calculations from 5-1) to 5-2) on entity pairs that do not have any relationship, and make predictions and judgments as independent spaces.
[0111] Step 6) The query set obtained in step 1) is processed from step 2) to step 4) to obtain the representations of all entity pairs to be predicted, and the Euclidean distance is calculated with the center of each type of relationship cluster in the metric space to make predictions.
[0112] 6-1) Calculation of likelihood function:
[0113] Get the entity pair concatenation r to be predicted in the query set i,j , let it be instance x, then the likelihood of it having relation y is calculated as follows:
[0114]
[0115] Among them, S is the support set, d is the Euclidean distance, m is the boundary threshold based on data distribution, and R is the relationship set in the task.
[0116] The calculation formula of Euclidean distance is:
[0117]
[0118] 6-2) Calculation of loss function:
[0119] By calculating the likelihood function in formula 5-1), we can obtain the predictions of entities i and j in the sentence for each relationship in R, and use the cross entropy function to calculate the loss:
[0120]
[0121] r′ i,j,k =p(y=k|x,S)
[0122]
[0123] where r′ i,j,k is the predicted probability that there is a relationship k between entity i and entity j, r i,j,k is the true label of the relationship k between entity i and entity j, and Q is the set of all entity pairs in the sentences of the query set.
[0124] Step 7) Run a large number of meta-tasks, repeat steps 2) to 6), train the network by gradient descent, and update the parameters.
[0125] Step 8) No parameter update is required in the test phase. The support set data in the meta-task of the test phase can be used to construct a metric space and predict the query set. That is, the support set in the meta-task is executed from step 2) to step 5), the query set is executed from step 2) to step 4), and then step 6) is executed to obtain the prediction result. Finally, the accuracy is evaluated.
[0126] The calculation formula for the average accuracy of all relations is as follows:
[0127]
[0128] Among them, acc i is the prediction accuracy of entity pairs with relation i.
[0129] The present invention adopts a few-sample relation extraction method based on metric learning, which can solve the situation where there are multiple entities in the text, the entities may be nested entities (such as the Chinese Navy and China are in a nested relationship), and there may be multiple relation triplets. In addition, by introducing adaptive intervals and adjusting the confidence of the categories in the support set, the prediction probability of categories with more data distribution can be increased, and the prediction probability of categories with less data distribution can be reduced, further improving the effect of the model.
[0130] In summary, the present invention designs a meta-task construction method for complex scenarios containing multiple entities and multiple relationships in the text, introduces adaptive intervals based on data distribution, and uses this to adjust the boundaries of the metric space. At the same time, it introduces a negative sample mechanism to solve the None-of-the-above scenario problem, and can also handle common entity nesting situations.
Claims
1. A method for extracting complex relations from a small number of samples based on adaptive intervals of data distribution, characterized in that: By constructing meta-tasks, complex relationships are extracted based on data distribution adaptive intervals and negative sample mechanisms; the following steps are included: Step 1: Divide the training data containing text instances and relation triplets into a series of meta-tasks in an N-way k-shot manner and store them in the task list [Task0, Task1, …]. Each meta-task contains a support set and a query set, and the ratio of the number of instances in the support set to the number of instances in the query set is 2:1 to 3:
1. Step 2: construct a pre-trained network for the encoding phase, process the data in each meta-task, and obtain the encoding of the text; Step 3: Encode the text of the support set / query set obtained in step 1 to obtain a vector representation; take the vectors of the corresponding positions of the entities in the current instance, aggregate them, and obtain the vector representation of each entity; Step 4: Combine and concatenate the entity vector representations in the support set / query set in pairs to obtain [e i ,e j ], i and j are entity serial numbers, i≠j; Step 5: The support set / query set obtained in step 1 contains N+ relation categories [R1, R2, …, R N+ ], the entity pair representations obtained in step 4 are stored according to the corresponding relationship. If there is no relationship between the entity pairs, they are stored as the None category, and the mean of the entity pair representations under each type of relationship is obtained as the vector cluster center of each type of relationship, that is, the vector representation of each type of relationship, thereby obtaining the metric space; the specific method is: 5-1) For each relationship classification, combine the entity pair representations contained in the classification to generate the cluster center of each relationship classification as the vector representation of the relationship; the vector representation of relationship k The calculation method is as follows, Among them, S kr is the concatenated set of entity pair vectors that have relation k; 5-2) Calculate the proportion of entity pairs corresponding to each type of relationship in each meta-task, and apply the dividing boundary to the relationship prediction to increase the probability of predicting the correct relationship label. For relationship k, its boundary threshold is defined as follows: Among them, |S k | and |S R \k|respectively represents the number of instance triples of relation k in the support set and the number of instance triples other than relation k, and R represents the relation set of the support set; 5-3) Set up a negative sample mechanism to perform calculations from 5-1) to 5-2) on entity pairs that do not have a relationship, and make predictions and judgments as independent spaces; Step 6, calculating the Euclidean distance between the representations of all entity pairs to be predicted in the support set and the query set obtained in step 4 and the center of each type of relationship cluster in the metric space, and performing relationship category prediction through an activation function; Step 7: Run a large number of meta-tasks, repeat steps 2 to 6, train the pre-trained network in the encoding phase by gradient descent, and update the network parameters.
Citation Information
Patent Citations
Entity relationship representation method and system for smart city few-sample data modeling
CN111858953A
Batch entity extraction method and system for professional domain papers
CN113051366A
Knowledge graph construction method based on adaptive few-sample relation extraction
CN113254675A
Neural network relationship extraction method, computer device, and readable storage medium
WO2021174774A1
