A relation extraction method fusing prototype knowledge

Abstract

The application discloses a relation extraction method fusing prototype knowledge, wherein in the input layer, input training samples; in the coding layer, a pre-training language model is used to mark the head and tail entities involved in the relation by symbols, and the hidden layer vectors corresponding to the marks are concatenated as a sentence coding vector; in the prototype vector layer, prototype vectors of each relation are first initialized, then through contrast learning, the distance between the sample and the prototype vector of the corresponding relation is minimized, the distance between the sample and the prototype vectors of other relation categories is maximized, the prototype vectors of each relation are learned and updated, and prototype knowledge is obtained; in the memory fusion layer, the prototype knowledge is fused into the coding vector of the current sample through a multi-head self-attention method, and is used for feature enhancement; in the output layer, a multi-classifier is used to classify and predict the relation between entities according to the sentence coding vector fused with the prototype knowledge.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing technology in artificial intelligence, and relates to a relation extraction method that integrates prototype knowledge. Background Technology

[0002] Relation extraction is a key task in natural language processing, serving a range of downstream applications such as knowledge graph construction and information retrieval. This task is defined as selecting appropriate relation category labels for candidate entity pairs from a set of candidate relation categories, based on the contextual description of the pair in text. Recent advancements in deep learning have sparked interest in supervised learning for relation extraction using neural networks. However, these methods require a substantial amount of labeled training data. Unfortunately, this data often exhibits a long tail, meaning that training samples for some relation categories are extremely scarce, leading to inaccurate model identification of these categories.

[0003] To alleviate this problem, supervised learning methods addressing class imbalance have been proposed. Early research in this area primarily focused on resampling or reassigning training data. Resampling methods reduce the sampling frequency of high-frequency classes and increase the sampling frequency of low-frequency relationships; reassignment methods assign smaller weights to high-frequency classes to reduce their impact on model optimization, and larger weights to low-frequency classes to increase their impact on model optimization. Unlike these methods, this proposal suggests a novel supervised learning approach for class imbalance that learns prototype knowledge to transfer information from high-frequency relationships to low-frequency relationships.

[0004] In class-imbalanced supervised learning problems, the data distribution exhibits a long tail phenomenon, with most classes containing only a small number of samples. This significantly impacts the overall performance and generalizability of the model. Therefore, it is necessary to propose a relation extraction method that integrates prototype knowledge to address this problem.

[0005] Previous research on relation extraction has largely focused on improving overall performance, neglecting the long-tail relation categories. Therefore, we observed room for improvement in relation extraction, focusing on the long-tail phenomenon. Previous methods primarily improved the model's optimization, mainly through training strategies such as data sampling and category assignment. They didn't address the inherent features of the data itself, failing to learn more discriminative feature vectors between categories through contrastive learning. However, for an effective relation extraction model, learning the most discriminative feature vectors for each relation category is crucial for accurate relation extraction.

[0006] Furthermore, previous models did not consider using prototype knowledge, thus failing to achieve feature fusion between memory features and current instances, nor the transfer of high-frequency features to low-frequency features. In some high-frequency relationships, features may be more universal, meaning they express low-frequency relationship features to some extent. If these features are fused into low-frequency instances, feature transfer can be achieved, thereby enhancing the features of low-frequency relationship categories. Nevertheless, previous relationship extraction research has largely neglected this feature transfer approach. Summary of the Invention

[0007] In view of this, the purpose of this invention is to propose a relation extraction method that integrates prototype knowledge. This method proposes a prototype vector training method based on contrastive learning and a prototype knowledge fusion method based on multi-head self-attention. First, after obtaining the original feature vectors of the samples, contrastive learning is used to narrow the distance between sentences with the same semantics and their prototypes in the vector space, while widening the distance between sentences with different semantics and their prototypes, thus obtaining prototype vectors with discriminative and representative characteristics. Second, after obtaining the prototype vectors, a multi-head self-attention method is used to obtain the prototype features most relevant to the sentence feature vectors from multiple feature perspectives. A feature combiner adaptively combines prototype knowledge with the sentence feature vectors to enhance the sentence's feature vectors. Finally, a multi-classifier predicts the relationships between entities involved in the sentence, and the method parameters are optimized using cross-entropy loss.

[0008] To achieve the above objectives, a relation extraction method integrating prototype knowledge is proposed, comprising the following steps:

[0009] Step 1: In the input layer, input the training samples;

[0010] Step 2: In the encoding layer, a pre-trained language model is used to mark the head and tail entities involved in the relation with symbols, and the hidden layer vectors corresponding to the marks are concatenated to form the sentence encoding vector.

[0011] Step 3: In the prototype vector layer, the prototype vectors of each relation are first initialized. Then, through comparative learning, the distance between the sample and the prototype vector of its corresponding relation is minimized, and the distance between the sample and the prototype vectors of other relation categories is maximized. The prototype vectors of each relation are then updated to obtain prototype knowledge.

[0012] Step 4: In the memory fusion layer, prototype knowledge is fused into the encoding vector of the current sample using a multi-head self-attention method for feature enhancement.

[0013] Step 5: In the output layer, a multi-classifier is used to classify and predict the relationships between entities based on sentence encoding vectors that incorporate prototype knowledge.

[0014] Specifically, in step 2, the input text sentence is appended with markers [E1], [ / E1], [E2], and [ / E2] before and after the beginning and end entities, respectively, to represent the start and end of the beginning entity and the beginning and end of the end entity. The sentence is represented as:

[0015] s′={[CLS],w1,...,w i-2 [E1], w i , ..., w j ,[ / E1],...,[E2],w k , ..., w l ,[ / E2],...,w n [SEP]

[0016] Wherein, the symbols [CLS] and [SEP] indicate the beginning and end of a sentence, w i Representing the i-th word of the text, the sentence is input into a pre-trained language model to obtain the hidden layer vector sequence of the sentence:

[0017]

[0018] The hidden layer vectors corresponding to identifiers [E1] and [E2] are concatenated and input into a fully connected layer, h. [CLS] h [SEP] Indicates the hidden layer state corresponding to characters [CLS] and [SEP], h i Indicator w i The corresponding hidden layer state, Representing the hidden layer states corresponding to the start and end markers of the beginning and end entities, we obtain the final sentence feature vector:

[0019]

[0020] Where W and b are trainable parameters, [:] represents a concatenation operation, and the vector dimension of the feature vector s is set to d.

[0021] In step 3, the prototype vector layer first initializes a prototype vector set P = {p1, p2, ..., p...} n}, where p i Let p represent the prototype knowledge of the i-th relation category, and there are n relation categories in total. Then, the prototype vector set is updated through comparative learning. The update principle is that the prototype vector p of each relation... i In the vector space, it should be similar to the feature vectors of all sentences in its category and far away from the feature vectors of sentences in other categories.

[0022] Specifically, firstly, a training sample set is constructed, given a set of sentence feature vectors {s1, s2, ..., s...}. B}, where B represents the cardinality of a set of sentence feature vectors. The training sample set consists of several sentence-prototype pairs. For a sentence feature vector s belonging to relation category j, i Then its positive example sentence-prototype pair is (s i ,p j The negative example sentence-prototype sample pair is represented as {(s)}. i ,p r )|1≤r≤n,r≠j}.

[0023] Next, the prototype vector is updated and optimized. The basic idea is to minimize the positive example distance and maximize the negative example distance in the training sample pairs in the vector space. The following loss function is defined:

[0024]

[0025] Where d(,) represents the distance function, defined as:

[0026]

[0027] In step 4, the memory fusion layer uses a multi-head self-attention mechanism to fuse the prototype vector as memory knowledge into the current sentence feature vector.

[0028] First, given M attention heads, perform self-attention operations on the sentence feature vector s to obtain:

[0029]

[0030] Where, q i K represents the linear transformation of the eigenvector s. i and V i For the linear transformation of the prototype vector set P, in order to obtain relevant information from multiple perspectives, the hidden features obtained from multi-head attention are converged to obtain:

[0031]

[0032] Where W1 are trainable parameters, x represents the feature vector converged after the self-attention layer operation. i Let represent the feature vector after the operation of the i-th attention head, M represent the number of attention heads, and LayerNorm(·) is a normalization operation function.

[0033] Finally, a feature combiner is used to fuse the prototype knowledge with the current sentence feature vector. The feature combiner consists of a fully connected layer, and the combination strength of the two features is dynamically adjusted through parameter learning. The fusion process is represented as follows:

[0034]

[0035] Where W2 and b2 are trainable parameters, This represents the sentence feature vector that incorporates prototype knowledge.

[0036] In this way, prototype knowledge can be used as a memory feature to enhance the features of the currently observed sentence, thereby improving the accuracy of relation extraction.

[0037] In step 5, the output layer uses the sentence feature vector that incorporates prototype knowledge. The relationship category is determined using a multi-classifier:

[0038]

[0039] Among them, W o and b o These are the trainable parameters, and Θ is the set of all trainable parameters. express The probability of belonging to category r. The r with the highest probability value is... The category.

[0040] Furthermore, the loss function in the output layer is defined as the cross-entropy loss function:

[0041]

[0042] Where, I(r=r) * () represents an indicator function, when r is The true category r * It is 1 if it is true, otherwise it is 0. Attached Figure Description

[0043] Figure 1 This is a flowchart illustrating an embodiment of the present invention;

[0044] Figure 2 This is a schematic diagram of the overall model structure of an embodiment of the present invention. Detailed Implementation

[0045] The present invention will be further described below with reference to the accompanying drawings, but this is not intended to limit the present invention in any way. Any modifications or substitutions made based on the teachings of the present invention shall fall within the protection scope of the present invention.

[0046] like Figure 1 As shown, the few-order text classification method under the metric-based meta-learning framework includes the following steps:

[0047] Step 1: In the input layer, input the training samples;

[0048] Step 2: In the encoding layer, a pre-trained language model is used to mark the head and tail entities involved in the relation with symbols, and the hidden layer vectors corresponding to the marks are concatenated to form the sentence encoding vector.

[0049] Step 3: In the prototype vector layer, the prototype vectors of each relation are first initialized. Then, through comparative learning, the distance between the sample and the prototype vector of its corresponding relation is minimized, and the distance between the sample and the prototype vectors of other relation categories is maximized. The prototype vectors of each relation are then updated to obtain prototype knowledge.

[0050] Step 4: In the memory fusion layer, prototype knowledge is fused into the encoding vector of the current sample using a multi-head self-attention method for feature enhancement.

[0051] Step 5: In the output layer, a multi-classifier is used to classify and predict the relationships between entities based on sentence encoding vectors that incorporate prototype knowledge.

[0052] A schematic diagram of the overall model structure of this invention embodiment is shown below. Figure 2 As shown.

[0053] Specifically, in step 2, the input text sentence is appended with markers [E1], [ / E1], [E2], and [ / E2] before and after its beginning and end entities, respectively, to represent the start and end of the beginning entity and the beginning and end of the end entity. The sentence can be represented as:

[0054] s′={[CLS],w1,...,w i-2 [E1], w i , ..., w j ,[ / E1],...,[E2],w k , ..., w l ,[ / E2],...,w n [SEP]

[0055] Wherein, the symbols [CLS] and [SEP] indicate the beginning and end of a sentence, w i Representing the i-th word of the text, the sentence is input into a pre-trained language model to obtain the hidden layer vector sequence of the sentence:

[0056]

[0057] The hidden layer vectors corresponding to identifiers [E1] and [E2] are concatenated and input into a fully connected layer, h. [CLS] h [SEP] Indicates the hidden layer state corresponding to characters [CLS] and [SEP], h i Indicator w i The corresponding hidden layer state, Representing the hidden layer states corresponding to the start and end markers of the beginning and end entities, we obtain the final sentence feature vector:

[0058]

[0059] Where W and b are trainable parameters, [:] represents a concatenation operation, and the vector dimension of the feature vector s is set to d.

[0060] In step 3, the prototype vector layer first initializes a prototype vector set P = {p1, p2, ..., p...} n}, where p i Let p represent the prototype knowledge of the i-th relation category, and there are n relation categories in total. Then, the prototype vector set is updated through contrastive learning to better represent the semantic knowledge of each relation. The principle of the update is that the prototype vector p of each relation... i In the vector space, it should be similar to the feature vectors of all sentences in its category and far away from the feature vectors of sentences in other categories.

[0061] Specifically, firstly, a training sample set is constructed, given a set of sentence feature vectors {s1, s2, ..., s...}. B Let B represent the cardinality of a set of sentence feature vectors. This training sample set consists of several sentence-prototype pairs. For a sentence feature vector s belonging to relation category j, ... i Then its positive example sentence-prototype pair is (s i ,p j The negative example sentence-prototype sample pair is represented as {(s)}. i ,p r )|1≤r≤n,r≠j}.

[0062] Next, the prototype vector is updated and optimized. The basic idea is to minimize the positive example distance and maximize the negative example distance in the training sample pairs in the vector space. The following loss function can be defined:

[0063]

[0064] Where d(,) represents the distance function, which is defined as:

[0065]

[0066] In step 4, the memory fusion layer uses a multi-head self-attention mechanism to fuse the prototype vector as memory knowledge into the current sentence feature vector.

[0067] First, given M attention heads, perform self-attention operations on the sentence feature vector s to obtain:

[0068]

[0069] Where, q i K represents the linear transformation of the eigenvector s. i and V i This represents a linear transformation of the prototype vector set P. To obtain relevant information from multiple perspectives, the hidden features obtained through multi-head attention can be aggregated to obtain:

[0070]

[0071] Where W1 are trainable parameters, x represents the feature vector converged after the self-attention layer operation. i Let represent the feature vector after the operation of the i-th attention head, and M represent the number of attention heads. LayerNorm(·) is a normalization operation function.

[0072] Finally, a feature combiner is used to fuse the prototype knowledge with the current sentence feature vector. The feature combiner consists of a fully connected layer, and the combination strength of the two features is dynamically adjusted through parameter learning. The fusion process is represented as follows:

[0073]

[0074] Where W2 and b2 are trainable parameters, This represents the sentence feature vector that incorporates prototype knowledge.

[0075] In this way, prototype knowledge can be used as a memory feature to enhance the features of the currently observed sentence, thereby improving the accuracy of relation extraction.

[0076] In step 5, the output layer uses the sentence feature vector that incorporates prototype knowledge. The relationship category is determined using a multi-classifier:

[0077]

[0078] Among them, W o and b o These are the trainable parameters, and Θ is the set of all trainable parameters. express The probability of belonging to category r. The r with the highest probability value is... The category.

[0079] Furthermore, the loss function in the output layer is defined as the cross-entropy loss function:

[0080]

[0081] Where, I(r=r) * () represents an indicator function, when r is The true category r * The value is 1 if the condition is met, and 0 otherwise. The loss is minimized using stochastic gradient descent.

[0082] The above embodiments are one implementation of the method of the present invention, but the implementation of the present invention is not limited to the embodiments described above. Any changes, modifications, substitutions, combinations, or simplifications made that deviate from the spirit and principle of the present invention should be considered equivalent substitutions and are included within the protection scope of the present invention.

Claims

1. A relation extraction method integrating prototype knowledge, characterized in that, Includes the following steps: Step 1: In the input layer, input the training samples; Step 2: In the encoding layer, a pre-trained language model is used to mark the head and tail entities involved in the relation with symbols, and the hidden layer vectors corresponding to the marks are concatenated to form the sentence encoding vector. Step 3: In the prototype vector layer, the prototype vectors of each relation are first initialized. Then, through comparative learning, the distance between the sample and the prototype vector of its corresponding relation is minimized, and the distance between the sample and the prototype vectors of other relation categories is maximized. The prototype vectors of each relation are then updated to obtain prototype knowledge. Step 4: In the memory fusion layer, prototype knowledge is fused into the encoding vector of the current sample using a multi-head self-attention method for feature enhancement. Step 5: In the output layer, a multi-classifier is used to classify and predict the relationships between entities based on the sentence encoding vectors that incorporate prototype knowledge. In step 4, the memory fusion layer uses a multi-head self-attention mechanism to fuse the prototype vector as memory knowledge into the current sentence feature vector; First, there is a setting Each attention head is used to focus on the sentence feature vector. Performing self-attention operations yields: in, For feature vectors linear change, and For the set of prototype vectors To obtain relevant information from multiple perspectives, the hidden features obtained from multi-head attention are converged to obtain the following linear transformation: in, For trainable parameters, This represents the feature vector after convergence through the self-attention layer operation. Let M represent the feature vector after the operation of the i-th attention head, and M represent the number of attention heads. This is a normalization operation function; Finally, a feature combiner is used to fuse the prototype knowledge with the current sentence feature vector. The feature combiner consists of a fully connected layer, and the combination strength of the two features is dynamically adjusted through parameter learning. The fusion process is represented as follows: in, and For trainable parameters, This represents the sentence feature vector that incorporates prototype knowledge.

2. The relation extraction method integrating prototype knowledge according to claim 1, characterized in that, In step 2, markers are added before and after the beginning and end entities of the input text sentence. , , , Let these represent the beginning and end of the header entity and the beginning and end of the tail entity, respectively. The sentence is represented as: Among them, symbols and Indicates the beginning and end of a sentence. The first character of the text Each word is input into a pre-trained language model to obtain the hidden layer vector sequence of the sentence: identifier and The corresponding hidden layer vectors are concatenated and input into a fully connected layer. , Character and The corresponding hidden layer state, Indicator The corresponding hidden layer state, , , , Representing the hidden layer states corresponding to the start and end markers of the beginning and end entities, we obtain the final sentence feature vector: in, and These are trainable parameters. This represents a concatenation operation, specifying the feature vector. The vector dimension is .

3. The relation extraction method integrating prototype knowledge according to claim 2, characterized in that, In step 3, the prototype vector layer first initializes a set of prototype vectors. ,in Indicates the first Prototype knowledge of 1 relation category, totaling There are several relation categories; then, the prototype vector set is updated through comparative learning, with the principle being that the prototype vector of each relation... In the vector space, it should be similar to the feature vectors of all sentences in its category and far away from the feature vectors of sentences in other categories; First, construct a training sample set, given a set of sentence feature vectors. B represents the cardinality of a set of sentence feature vectors. The training sample set consists of several sentence-prototype pairs. For a sentence belonging to the relation category... Sentence feature vectors Then its positive example sentence-prototype pair is The negative example sentence-prototype sample pair is represented as: ; Next, the prototype vector is updated and optimized. The basic idea is to minimize the positive example distance and maximize the negative example distance in the training sample pairs in the vector space. The following loss function is defined: in, The distance function is defined as follows: 。 4. The relation extraction method integrating prototype knowledge according to claim 3, characterized in that, In step 5, the output layer uses the sentence feature vector that incorporates prototype knowledge. The relationship category is determined using a multi-classifier: in, and These are trainable parameters. For the set of all training parameters, express Category The probability of.

5. The relation extraction method integrating prototype knowledge according to claim 4, characterized in that, Define the cross-entropy loss function in the output layer: in, Represents an indicator function, when for Real category It is 1 if it is true, otherwise it is 0.

Images