A small sample relation classification method and device based on a global transformation prototype network

Abstract

The application discloses a kind of based on global transformation prototype network's small sample relation classification method and device, belong to natural language processing technical field.The method mainly includes: (one) based on pre-training language model and entity mark's relation instance representation extraction mechanism;(two) based on global transformation prototype network of relation instance representation;(three) based on the global prototype representation obtained by the above network, the small sample relation classification method of being carried out.The pre-training language model and entity mark's relation instance representation extraction mechanism of the application can be fully encoded to the entity pair and relevant context included in relation instance;Based on global transformation's prototype network can be fully obtained by support set Global feature of new task, based on this, new relation prototype representation is transformed, so that better domain generalization is carried out.

Description

TECHNICAL FIELD

[0001] The application relates to a small sample relation classification method, in particular to a small sample relation classification method and device based on a global transformation prototype network, and belongs to the technical field of natural language processing. BACKGROUND

[0002] Relation extraction is a key task in information extraction and natural language processing. Relation classification is a key step in relation extraction. The goal of relation classification is to identify the relation category expressed by a pair of entities in a given context. For example, given the context "A is elected as the president of B", and the entity pair "A" and "B", a relation classification system should be able to correctly identify that the entity pair expresses the "is… president" relation.

[0003] In recent years, with the development of deep learning methods, the accuracy of relation classification systems has been greatly improved. However, this method requires a large amount of labeled data, resulting in high labeling costs for training models. At the same time, the model obtained by this method has poor generalization and cannot be migrated to new relations and domains.

[0004] Therefore, small sample relation classification tasks have received widespread attention in recent years. This task assumes that each new relation classification task is divided into a support set and a query set, each support set contains N new relation categories, and each category contains only K relation instances (K is usually 1 or 5). The model needs to rely on only N*K labeled instances to complete the relation classification of unlabeled instances in the query set.

[0005] Traditional small sample relation classification methods are mainly based on metric learning. This method encodes relation instances into a fixed semantic space and uses labeled instances in the support set to calculate the representation prototype of each relation. For each unlabeled instance in the query set, it is classified into the nearest relation representation prototype, thereby predicting the corresponding relation category.

[0006] However, the metric-based method assumes that the metric space distributed by each new relation classification task is consistent, which leads to the inability of the method to effectively distinguish between semantically similar relations in new tasks, and the inability to effectively generalize in cross-domain scenarios. SUMMARY

[0007] To overcome the problem that existing small sample relation classification methods based on metric learning cannot effectively generalize in cross-domain tasks, the application provides a small sample relation classification method and device based on a global transformation prototype network.

[0008] The main contents of the application include: (1) a relationship instance representation extraction mechanism based on a pre-trained language model and entity tagging; (2) a global transformation prototype network based on relationship instance representation; and (3) a small sample relationship classification method based on the global prototype representation obtained by the network.

[0009] The technical solutions adopted by the application are as follows:

[0010] A small sample relationship classification method based on a global transformation prototype network includes the following steps:

[0011] The relationship instances in the support set are tagged using a tag symbol, and the relationship instance representation is obtained through a pre-trained language model;

[0012] Based on the relationship instance representation, the global transformation prototype network is used to uniformly transform the relationship instances in the support set to obtain the relationship prototype representation;

[0013] Based on the relationship prototype representation, the small sample relationship classification is performed on the relationship instances in the query set.

[0014] Further, the use of a tag symbol to tag the relationship instances in the support set and the obtaining of the relationship instance representation through a pre-trained language model includes:

[0015] The entity pairs in the sentence are tagged with special tag symbols, and the tag symbols are set before and after each entity in the entity pair to mark the boundaries thereof;

[0016] The tagged sentence is input into a pre-trained language model to convert all characters in the sentence into semantic vectors;

[0017] The semantic vectors corresponding to the special tag symbols in the sentence are extracted and spliced to form the final relationship instance representation.

[0018] Further, the pre-trained language model is a Bert model or a RoBERTa model.

[0019] Further, the use of a global transformation prototype network based on the relationship instance representation to uniformly transform the relationship instances in the support set to obtain the relationship prototype representation includes:

[0020] All relationship instance representations in the support set are combined to form a support sequence;

[0021] For the support sequence, three different global transformation encoders are used to transform from the intra-class relationship, the inter-class relationship, and the global perspective, respectively;

[0022] The representations output by the three different global transformation encoders are spliced to form the output of the final global transformation prototype network, and the relationship prototype representation is obtained.

[0023] Further, the small sample relation classification based on the relation prototype representation comprises:

[0024] Calculate the distance between each relation instance in the query set and the relation prototype representation, and calculate the classification probability of each relation class after normalization;

[0025] In the training phase, the cross-entropy loss of the classification probability and the actual probability is calculated, and the training model is updated by back propagation;

[0026] In the prediction phase, the relation with the maximum probability is directly used as the prediction result output.

[0027] A small sample relation classification device based on the global transformation prototype network using the above method, comprising:

[0028] A relation instance representation acquisition module is used to mark the relation instances in the support set using a token, and obtain the relation instance representation through a pre-trained language model;

[0029] A relation prototype representation acquisition module is used to uniformly transform the relation instances in the support set based on the relation instance representation using the global transformation prototype network to obtain the relation prototype representation;

[0030] A small sample relation classification module is used to classify the relation instances in the query set based on the relation prototype representation.

[0031] The beneficial effects of the present application are:

[0032] The relation instance representation extraction mechanism based on the pre-trained language model and entity marking of the present application can fully encode the entity pairs and related context included in the relation instance, the prototype network based on global transformation can fully obtain the global features of the new task through the support set, and the new relation prototype representation is transformed based on this, so that the domain generalization can be better performed. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 It is the overall architecture of the global transformation prototype network. Wherein, q is the query instance representation to be classified, Represent the jth relation instance of the ith relation class in the support set, and "N x" represents N identical modules stacked. DETAILED DESCRIPTION

[0034] The present application will be further described in detail below through specific embodiments and drawings.

[0035] The key steps of the small sample relation classification method based on the global transformation prototype network of the present application include:

[0036] 1) The relationship instance representation extraction mechanism based on the pre-trained language model and entity marking uses the pre-trained language model Bert to provide more rich context-related representation, and adopts entity marking to make the final relationship instance representation pay more attention to entity information.

[0037] 2) Based on the above relationship instance representation, the global transformation prototype network is used to uniformly transform the relationship instances in the support set to obtain the prototype representation of the new relationship.

[0038] 3) Based on the relationship prototype representation obtained by the above global transformation prototype network, small sample relationship classification is carried out, so as to have more generalization in cross-domain and other scenes.

[0039] The present application relates to the following key elements:

[0040] 1. Relationship instance representation based on pre-trained language model and entity marking

[0041] In the current natural language processing task. The context-related representation features brought by the pre-trained language model have high generalization, which greatly improves the performance of many natural language processing tasks, and has become the basic module of many tasks. The relationship classification task has its own characteristics, that is, only the semantic features related to the current entity pair are concerned, so as to classify the relationship. In order to better utilize the semantic features in the pre-trained language model, and to meet the entity pair-centered semantic representation in relationship classification, the entity marking method is adopted.

[0042] Specifically, given a sentence containing n words: X=[x0,…,x n-1 ], and a pair of entities contained therein: e1=[i,j],e2=[k,l],0≤i≤j<k≤l≤n-1. A relationship instance can be represented as: (X,e1,e2). For example, ("A company chief executive officer B", [0,3], [9,10]) is a relationship instance, and expresses the relationship of "chief executive officer". In order to convert discrete characters into continuous vectors that can be processed by neural networks, the relationship instance needs to be encoded into relationship representation, including the following steps:

[0043] 1) First, mark the entity pair with special marking symbols. The marking method is: set marking symbols before and after each entity of the entity pair to mark its boundary. The above example is processed as: ("A <e1>Company A< / e1> chief executive officer <e2> B< / e2> ", [0,5], [11,13]), wherein <e1> 、< / e1> 、 <e2> 、< / e2> is a special marking symbol, <e1>before the first entity of the entity pair,< / e1> is placed after the first entity of the entity pair, <e2>before the second entity of the entity pair,< / e2> is placed after the second entity of the entity pair.

[0044] 2) Then, the processed sentence is sent into a pre-trained language model, and all the characters in the sentence are converted into semantic vectors. The pre-trained language model can adopt a model such as Bert (Bidirectional Encoder Representations from Transformers), RoBERTa (Robustly Optimized BERT Pretraining Approach), etc.

[0045] 3) In order to form the final relationship instance representation vector, the special marker symbol "##sep##" in the sentence is removed, and the semantic vectors of the words in the sentence are concatenated to form the final relationship instance representation vector. <e1>"and" <e2>The corresponding encoded vector (i.e., the semantic vector obtained from the pre-trained language model in step 2) is extracted and concatenated to form the final representation vector h∈R of the relation instance. d Where d is the dimension of the instance representation vector.

[0046] 2. Global Transform Prototype Network Based on Instance-Level Representation

[0047] Global transformation prototype networks based on instance-level representations can fully capture information supporting the intra-class, inter-class, and global relationships of relation sets, and use this information to transform and represent relation prototypes, making the transformed relation prototype representations more separable and better adaptable to cross-domain scenarios. For example... Figure 1 As shown, to model global information on the support set, all relation instance representations in the support set are taken as input, and a Transformer network is used to globally encode and transform these instance representations. Specifically, all support instances (all relation instances in the support set) are first combined to form a new support sequence X:

[0048]

[0049] in, The representation of the j-th relation instance of the i-th relation category in the support set. Let X represent the j-th relation instance of the i-th relation category in the support set. The support set contains N relation categories, and each relation category contains K relation instances. For simplicity, the support sequence X will be re-represented as follows:

[0050] X = [h] 11 ,…,h ij ,…,h NK ]

[0051] Among them, h ij This is to support the representation of the i-th relation and j-th instance in sequence X.

[0052] After processing the input, we implemented three different global transform encoders to perform transformations from intra-relational, inter-relational, and global perspectives, respectively. Specifically, we first input the sequence containing all supporting instances into the Transformer network for the first step of multi-view mapping;

[0053]

[0054] Among them, h ij W is the j-th instance of the i-th relation in X. k and b k is the mapping matrix and bias of the kth view (a potential semantic view, realized by multi-head mechanism in Transformer), is h ij is the representation of the kth view. After multi-view mapping, we use self-attention mechanism for each view respectively, and use the mask matrix customized by different global transformation encoders to control the area of self-attention focus:

[0055]

[0056]

[0057] where M ijmn is the mask between the i-th instance of the j-th relation and the m-th instance of the n-th relation in the mask matrix, with a value of 0 or -∞; α ijmn is the attention weight of the i-th instance of the j-th relation and the m-th instance of the n-th relation, is the representation vector of the kth view of the m-th instance of the n-th relation, d k is the dimension of the representation vector. The mask matrix of different transformation encoders is selected differently, as follows:

[0058] Intra-relation class encoder mask:

[0059]

[0060] Inter-relation class encoder mask:

[0061]

[0062] Global encoder mask:

[0063] M ijmn = 0

[0064] When M ijmn is 0, it means that the information interaction exists between the two instances; when M ijmn is -∞, it means that the i-th instance of the j-th relation and the m-th instance of the n-th relation are invisible.

[0065] Then the representations output by the three encoders are spliced to form the output of the final global transformation prototype network:

[0066]

[0067] where MLP represents multi-layer perception (Multi-layer perception), represents the vector output by the intra-class encoder after encoding the i-th instance of the j-th relation, represents the vector output by the inter-class encoder after encoding the i-th relationship j-th instance, represents the vector output by the global encoder after encoding the i-th relationship j-th instance.

[0068] After obtaining the transformed representations of all support instances, we use the global transformation prototype network to calculate the new relationship prototype representation:

[0069]

[0070] p i is the prototype representation of the i-th relationship class, h ij is the j-th instance of the i-th relationship in the support set, and K represents the number of instances included in the support set.

[0071] This step is based on the pre-trained language model and the entity-labeled relationship instance representation features, and all support instances are uniformly modeled for global transformation, so as to obtain the relationship prototype representation quickly adapted to the new task, thereby alleviating the problem that the original measurement method is difficult to effectively generalize to similar relationships and cross-domain relationship classification tasks, and obtaining more robust and more accurate generalization results.

[0072] 3. Small sample relationship classification based on global transformation prototype representation

[0073] After obtaining the prototype representation of each relationship class, we simply measure the similarity between the query instance (relationship instance in the query set) and each class prototype by the Euclidean distance, and normalize it by Softmax to obtain the classification probability:

[0074]

[0075] where D(.) is the distance function, y i is the prediction of the i-th class, S i is the support set of the i-th relationship, q is the query instance, and θ is the model parameter, h q is the representation of the query instance, and N is the total number of relationship classes.

[0076] After obtaining the classification probability, the relationship with the maximum probability is output as the prediction result.

[0077] Next, a 3-class 2-sample small sample relationship classification task is taken as an example to illustrate the method of the present application.

[0078] Support set:

[0079] The first class is the relationship between husband and wife:

[0080] 1. <e1>Wang Meng< / e1> is <e2>Zhang San< / e2> the wife of”;

[0081] 2. <e1>Li Hua< / e1> married to <e2>Zhao Si< / e2> ”.

[0082] Second category - parent relationship:

[0083] 1. <e1>Zhang San< / e1> is <e2>Li Si< / e2> ’s son”;

[0084] 2. <e1>Zhao Si< / e1> ’s daughter called <e2>Wang Meng< / e2> ”

[0085] Third category - sibling relationship:

[0086] 1. <e1>Zhang Kai< / e1> is <e2>Zhang Jun< / e2> ’s brother”;

[0087] 2. <e1>Zhang Jun< / e1> and <e2>Zhang Kai< / e2> ’s parent”;

[0088] Query set:

[0089] 1. <e1>Everyday< / e1> ’s wife named their two children <e2>Kevin< / e2> and ”

[0090] Implementation:

[0091] (I) For all support set instances, pre-process the entity pairs, mark them with token symbols, and concatenate the head and tail token vectors as the instance representation through a pre-trained language model.

[0092] (II) Based on the global transformation prototype network, model and transform the global information of the support set instances. For example, the representation of all six relationship instances of the three categories is concatenated into a sequence and input into the global transformation prototype network for automatic global information modeling and transformation. The output of the six instance representations is used as the transformed representation.

[0093] (III) Based on the transformed instance representation and the prototype network, calculate the relationship prototype representation of each category. Then calculate the distance between each instance in the query set and the prototype representation, and calculate the classification probability of each category after normalization. In the training phase, the cross-entropy loss between the classification probability and the actual probability is calculated, and the training model is updated through backpropagation. In the prediction phase, the relationship with the maximum probability is directly used as the prediction result output.

[0094] The effect of the method of the present application on the current internationally popular small sample relation extraction evaluation system FewRel is shown in Tables 1 and 2, wherein the FewRel1.0 task is a small sample relation classification task of the same field and different relations; and the FewRel 2.0 field adaptation task is a small sample relation classification task of different fields and different relations.

[0095] Table 1. FewRel 1.0 test results

[0096]

[0097] Table 2. FewRel 2.0 field adaptation task test results

[0098]

[0099] It can be seen that the method of the present application has good performance on the two tasks and greatly exceeds the comparative system in multiple indicators. Among them, the model of the present application compared with CP and MTB is the best model published at present.

[0100] In addition to the global transformation mode based on the transformer used in the above-mentioned embodiments, the global information can also be transformed by other graph neural networks (such as GNN, GCN, etc.).

[0101] Based on the same inventive concept, another embodiment of the present application provides a small sample relation classification device based on the global transformation prototype network using the above-mentioned method, which comprises:

[0102] A relation instance representation acquisition module is configured to mark the relation instances in the support set using a mark symbol and obtain the relation instance representation through a pre-trained language model.

[0103] A relation prototype representation acquisition module is configured to uniformly globally transform the relation instances in the support set based on the relation instance representation by using the global transformation prototype network to obtain the relation prototype representation.

[0104] A small sample relation classification module is configured to perform small sample relation classification on the relation instances in the query set based on the relation prototype representation.

[0105] Based on the same inventive concept, another embodiment of the present application provides an electronic device (computer, server, smart phone, etc.), which comprises a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing each step in the method of the present application.

[0106] Based on the same inventive concept, another embodiment of the present application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk) storing a computer program, which, when executed by a computer, implements each step of the method of the present application.

[0107] The specific embodiments of the application disclosed above are intended to be illustrative only. Changes can be made to the embodiments, by one skilled in the art, without departing from the spirit and scope of the application which is defined by the following claims. The present application should not be limited to the embodiments disclosed herein, but should be given the broadest possible interpretation within the lawful limits of the patent claims.

Claims

1. A small sample relation classification method based on a global transformation prototype network, characterized in that, Includes the following steps: The relation instances in the support set are labeled using symbolic notation, and the relation instance representations are obtained through a pre-trained language model; Based on relation instance representation, a global transformation prototype network is used to uniformly transform relation instances in the support set to obtain relation prototype representation; Based on relation prototype representation, perform small sample relation classification on relation instances in the query set; The relation instance-based representation utilizes a global transformation prototype network to uniformly transform relation instances in the support set, resulting in a relation prototype representation, including: All relation instances in the support set are represented as a support sequence X; For the supporting sequence X, three different global transform encoders are used to transform from the perspectives of intra-relation class, inter-relation class, and global, respectively. The representations output by the three different global transform encoders are concatenated to form the output of the final global transform prototype network, thereby obtaining the relation prototype representation; The three different global transform encoders use different mask matrix choices, as detailed below: Encoder mask within relation class: Inter-class encoder mask: Global encoder mask: M ijmn ＝0 When M ijmn is 0, it represents that the current two instances have information interaction with each other; when M ijmn is -∞, it represents that the ith relationship jth instance and the mth relationship nth instance are invisible.

2. The method of claim 1, wherein, The step of labeling relation instances in the support set with symbolic notation and obtaining relation instance representations through a pre-trained language model includes: The entity pairs in the sentence are marked with special markers. Marker symbols are set before and after each entity in the entity pair to mark its boundaries. The tagged sentence is fed into a pre-trained language model to convert all characters in the sentence into semantic vectors; Extract the semantic vectors corresponding to the special markers in the sentence, concatenate them to form the final relation instance representation.

3. The method of claim 1, wherein, The pre-trained language model is either a BERT model or a RoBERTa model.

4. The method of claim 1, wherein, The method of classifying small-sample relations based on relation prototype representation includes: Calculate the distance between each relation instance and the relation prototype representation in the query set, and calculate the classification probability of each type of relation after normalization; During the training phase, the cross-entropy loss between the classification probability and the actual probability is calculated, and the training model is updated through backpropagation. In the prediction phase, the relationship with the highest probability is used directly as the prediction result output.

5. The method of claim 4, wherein, The formula for calculating the classification probability is: Where D(.) is the distance function, y i To predict the i-th category, S i Let q be the support set for the i-th relation, θ be the query instance, and h be the model parameters. q For the representation of the query instance, N is the total number of relation categories.

6. A small sample relation classification device based on a global transformation prototype network using the method of any one of claims 1 to 5, characterized by, include: The relation instance representation acquisition module is used to label relation instances in the support set using symbolic notation and obtain relation instance representations through a pre-trained language model; The relation prototype representation acquisition module is used to perform a unified global transformation on relation instances in the support set based on the relation instance representation, and obtain the relation prototype representation by using a global transformation prototype network. The few-sample relation classification module is used to classify relation instances in the query set based on relation prototype representation.

7. An electronic device, comprising: It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method described in any one of claims 1 to 5.

Images