Incremental few-sample relation classification methods, devices, equipment, and storage media for fusion relation concepts

Abstract

This invention discloses an incremental few-shot relation extraction method, apparatus, device, and storage medium incorporating relation concepts. The method acquires a relation extraction dataset, partitions the dataset, and constructs an incremental few-shot relation extraction dataset; extracts sentence semantic information and entity pair semantic information; trains prototype embeddings of basic relations based on labeled samples of basic relations; fixes the encoder and prototype embeddings trained in the basic stage to facilitate incremental learning, and selects representative samples of basic relations; constructs multiple sets of different pseudo-incremental few-shot meta-learning tasks, and incorporates corresponding relation concepts into supporting instances of newly added relations; trains an incremental few-shot relation extraction model based on the constructed meta-learning tasks; incorporates relation prototypes into the feature representation of the query sentence, calculates the metric between the feature representation and the relation prototype vector, and achieves relation extraction from the query sentence. This invention enhances the model's feature vector representation of supporting instances of newly added relations in incremental few-shot scenarios, better adapting to relation extraction tasks in incremental few-shot scenarios and improving the accuracy of incremental few-shot relation extraction.

Description

Technical Field

[0001] This invention relates to the field of natural language processing in the field of information technology, and particularly to the technical fields of artificial intelligence, information extraction, incremental few-shot learning, and relation extraction. Specifically, this invention provides a method, apparatus, device, and storage medium for incremental few-shot relation extraction that integrates relation concepts. Background Technology

[0002] Relation extraction (Relation Classification) aims to extract the relationships between entity pairs from given text and is a crucial subtask of information extraction. Traditional relation extraction models lack the ability to incrementally learn new relations. Recently, few-shot relation extraction has attracted considerable interest and achieved satisfactory classification performance on new relations. However, these few-shot relation extraction models focus solely on identifying new relations, neglecting the learned foundational relation knowledge. Furthermore, most existing relation extraction models are based on closed-domain scenarios, where all target relations are predefined, limiting model flexibility. In real-world scenarios, the desired approach is for models to incrementally learn new relations with only a small number of labeled samples (1 or 5) of newly added relations, without forgetting the learned foundational relations.

[0003] This challenging yet practical task requires the model to recognize both basic and novel relationships. Incremental few-shot learning typically involves two training phases: a base learning phase and an incremental learning phase. First, in the base learning phase, a deep neural network is trained on a base class with many instances. Second, in the incremental learning phase, an incremental model is trained using metric learning and meta-learning strategies. Metric learning enables the model to better handle few-shot scenarios, while meta-learning provides the model with incremental learning capabilities by constructing pseudo-incremental tasks from existing samples.

[0004] Existing incremental few-sample relation extraction methods have the following problems:

[0005] 1) When learning new relations based on only a small number of labeled examples (1 or 5), there is a problem of overfitting to a small number of labeled examples. Unlike images, text is more abstract and may contain noise. When the labeled samples are too extreme to reflect the characteristics of the new relations, it will limit the model's ability to learn new relations.

[0006] 2) When the model incrementally learns new relations, a catastrophic forgetting problem occurs regarding the learned basic knowledge. When the model incrementally learns new relations, the basic relations and new relation samples are classified in the same embedding space. This can lead to the embeddings of basic relation instances deviating from their respective relation prototypes, resulting in a catastrophic forgetting problem.

[0007] To address the problems existing in the aforementioned incremental few-sample relation extraction methods, this invention provides an incremental few-sample relation extraction method, apparatus, device, and storage medium that integrates relation concepts. The technology of this application combines metric methods and contrastive learning. Based on a prototype network and meta-learning strategy, a relation concept injection module is added. By incorporating relation concepts into the sample representation, the overfitting problem for a small number of supporting samples is alleviated, generating a prototype representation in the feature space that better fits the new relation, thus improving the model's ability to learn new relations. Secondly, a contrastive learning construction module is added to construct positive and negative examples for basic relations. By using a contrastive loss function, the model mitigates the problem of forgetting old knowledge when learning new relations. Summary of the Invention

[0008] The present invention aims to provide an incremental few-shot relation extraction method, apparatus, device, and storage medium that integrates relation concepts. The method acquires a relation extraction dataset, partitions the dataset, and constructs an incremental few-shot relation extraction dataset; extracts sentence semantic information and entity pair semantic information; trains prototype embeddings of basic relations based on labeled samples of basic relations; fixes the encoder and prototype embeddings trained in the basic stage to facilitate incremental learning, and selects representative samples of basic relations; constructs multiple sets of different pseudo-incremental few-shot meta-learning tasks, and integrates corresponding relation concepts into supporting instances of newly added relations; trains an incremental few-shot relation extraction model based on the constructed meta-learning tasks; integrates relation prototypes into the feature representation of the query sentence, calculates the metric between the feature representation and the relation prototype vector, and achieves relation extraction from the query sentence. This invention enhances the model's feature vector representation of supporting instances of newly added relations in incremental few-shot scenarios, better adapts to relation extraction tasks in incremental few-shot scenarios, and improves the accuracy of incremental few-shot relation extraction.

[0009] The incremental few-sample relation extraction method for fusing relation concepts described in this invention is performed according to the following steps:

[0010] a. Obtain a few-sample relation extraction dataset, divide the dataset into basic relation, new relation validation set and new relation test set, and construct an incremental few-sample relation extraction dataset;

[0011] b. Based on labeled basic relation samples, special markers are introduced, and pre-trained language models are used to extract sentence semantic information and entity pair semantic information;

[0012] c. Train the prototype embedding of basic relations on labeled basic relation samples, train the basic relation extraction model based on the semantic feature vector of the training samples, and learn the prototype embedding representation of basic relations.

[0013] d. After fixing the encoder and prototype embedding trained in the basic stage, select representative samples of basic relationships based on the idea of ​​clustering;

[0014] e. Construct a pseudo-incremental few-sample meta-learning task based on the basic relation set, obtain the feature representation of relation concepts through the pre-trained MPNET model, and integrate the learning of new relations into the corresponding relation concepts.

[0015] f. Train the incremental few-sample relation extraction model based on the constructed meta-learning task;

[0016] g. Integrate the feature representation of the query sentence into the relation prototype, calculate the metric between the relation prototype vector and the feature representation, and extract the relation from the query sentence.

[0017] The construction of the incremental few-sample relation extraction dataset mentioned in step a is as follows: the entire dataset is divided into a basic relation set, a new relation validation set, and a new relation test set. The three sets have no overlap and each contains a training sample set, a validation sample set, and a test sample set. Each sample set has no overlap.

[0018] The extraction of sentence-level semantics in step b involves adding a special marker [CLS] at the beginning of the sentence and a special marker [SEP] at the end. The text is then context-encoded using a pre-trained BERT model, and the output at [CLS] is selected as the sentence-level semantic feature vector. The extraction of entity-pair semantics involves adding special markers [unused0], [unused1], [unused2], and [unused3] at the beginning and end of the text. The processed text is then input into the pre-trained BERT model. The byte-encoded representations of the beginning and end entity pairs are obtained using the auxiliary markers [unused0], [unused1], [unused2], and [unused3]. The average of these features is used as the entity-pair semantic information, which is then concatenated with the sentence-level semantic information to obtain the final semantic feature vector containing both entity pairs and the sentence.

[0019] The fixed encoder and prototype embedding mentioned in step d are as follows: the learned basic relation prototype embedding representation and its pre-trained BERT model used for feature extraction are fixed, and the parameter weights finely tuned in this task are retained; the representative samples of the basic relation are extracted as follows: based on the idea of ​​clustering, the top N sample embedding representations that are most similar to the relation prototype embedding are extracted from the training samples of the basic relation, and the average top N vector representations are fixed and used as the prototype embeddings of the basic relation in the incremental learning stage.

[0020] The pseudo-incremental few-sample meta-learning task described in step e is as follows: construct basic relations and new relations from the basic relation set, and construct a support set from the training set or test set of the basic relations; for the support instances of the new relations, the corresponding relation concept is integrated as follows: the semantic feature vectors obtained by the fixed BERT model from the few support instances 1 or 5 samples of the new relation are used as text features, and the vector representation of the relation concept is obtained by the pre-trained MPNET model. Then, the vector representation of the relation concept is fixed, and the embedding representation of the relation concept is integrated into the sentence-level semantic feature vector of the support instances through contrastive learning. Finally, the vector representations of the same relation are averaged to obtain the prototype representation of the new relation.

[0021] The training of the incremental few-shot relation extraction model in step f is as follows: the query sentence is compared with the prototype by calculating the metric between the query sentence and the prototype through the prototype network, and then classified. The incremental few-shot relation extraction model is trained based on the cross-entropy loss and the contrastive loss.

[0022] The feature representation of the query sentence in step g is integrated into the relation prototype by: assigning similarity weights based on the similarity between the semantic feature vector of the query sentence and the prototype embeddings of the basic and newly added relations, and generating a vector representation of the query sentence; the relation extraction of the query sentence is: calculating the distance between the vector representation of the query sentence and all relation prototype vectors, and using the nearest neighbor idea to extract the relation of the query sentence.

[0023] An incremental few-shot relation extraction device that integrates relation concepts is disclosed. This device comprises an incremental few-shot dataset construction module, a semantic encoder module, a basic relation extraction module, a relation concept injection module, a contrastive learning construction module, a query text feature generation module, and an incremental relation extraction module, wherein:

[0024] The incremental few-shot dataset construction module acquires the relation extraction dataset, processes the dataset, and constructs the incremental few-shot relation extraction dataset. Specifically, it includes a relation set splitting unit, used to divide the entire dataset into a base relation set and a new relation set, used to construct the incremental few-shot learning scenario; a dataset splitting unit, used to divide the entire dataset into a training set, a validation set, and a test set; and a pseudo-incremental meta-task construction unit, used to construct multiple different pseudo-incremental few-shot meta-learning tasks during training and testing.

[0025] The semantic encoder module encodes text into high-dimensional vector features. Specifically, it includes a semantic information extraction unit, which adds special markers [CLS] at the beginning of a sentence and [SEP] at the end, and obtains semantic representations through a pre-trained BERT model; a relation concept semantic extraction unit, which adds special markers [CLS] at the beginning of relation concepts, and obtains vector representations of relation concepts through a pre-trained MPNET model; and an entity pair semantic extraction unit, which adds special markers at the beginning and end of entities in a sentence, extracts the byte features of entity pairs, and calculates the average to obtain high-dimensional feature representations of the beginning and end entities.

[0026] Basic relation extraction module: Learns the prototype embedding representation of basic relations using the nearest neighbor concept by using labeled samples of the basic relation set.

[0027] Relationship concept injection module: For the newly obtained text representation of a relation, copy its sentence-level vector representation and pass it through a fully connected layer and activation function to construct a contrastive loss and incorporate its corresponding relation concept; to ensure the consistency of feature dimensions, copy the text representation of the basic relation and concatenate it with its sentence-level feature vector and the original semantic vector.

[0028] Contrastive learning construction module: Samples of newly added relations are constructed as positive examples, and samples of other new relations and basic relations are constructed as negative examples. The text feature vector representation that is easier to divide is obtained by contrastive loss.

[0029] Query text feature generation module: Based on the similarity between the semantic feature vector of the query sentence and the prototype embeddings of the basic and newly added relations, similarity weights are assigned to generate a vector representation of the query sentence;

[0030] Incremental relation extraction module: Calculates the distance between the vector representation of the query sentence and the prototype vectors of all relations, and extracts relations from the query sentence using the nearest neighbor concept.

[0031] An electronic device includes at least one processor; at least one GPU computing card; and at least one memory communicatively connected to the processor; wherein the memory stores instructions executable by the at least one processor or the at least one GPU computing card to enable the at least one processor or the at least one GPU computing card to perform the method.

[0032] A non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to cause the computer to perform the method.

[0033] This invention provides an incremental few-sample relation extraction method for fusing relation concepts, the method comprising:

[0034] Obtain the relation extraction dataset, partition the dataset, and construct an incremental few-sample relation extraction dataset;

[0035] Based on labeled basic relation samples, extract sentence semantic information and entity pair semantic information;

[0036] Train prototype embeddings of basic relations based on labeled samples of basic relations;

[0037] The encoder and prototype embedding trained in the fixed base phase are fixed, and representative samples of the base relations are selected.

[0038] Construct multiple sets of pseudo-incremental few-sample meta-learning tasks, and integrate the corresponding relation concepts into the support instances of newly added relations;

[0039] Based on the constructed meta-learning task, an incremental few-sample relation extraction model is trained.

[0040] The feature representation of the query sentence is incorporated into the relation prototype, and the metric between the feature representation and the relation prototype vector is calculated to extract the relation from the query sentence.

[0041] This invention provides an incremental few-sample relation extraction device, the device comprising:

[0042] Dataset building module: Obtains the relation extraction dataset, processes the dataset, and builds an incremental few-sample relation extraction dataset;

[0043] Semantic encoding module: Extracts semantic information from sentences and entity pairs, and encodes sentences into high-dimensional feature vectors;

[0044] Basic relation extraction module: Learns the prototype embedding representation of the basic relation based on the feature vectors of the basic relation text;

[0045] Relationship Concept Injection Module: For the feature representation of newly added relation support text, its corresponding relation concept is incorporated;

[0046] Contrastive learning constructs a module that constructs positive and negative examples of basic and newly added relations. These positive and negative examples are then used to obtain a prototype embedding representation of the newly added relations after the relation concept is integrated.

[0047] Query text feature generation module: Based on the similarity between the relation prototype and the high-dimensional feature representation of the query sentence, assign similarity weights and generate a vector representation of the query sentence;

[0048] Incremental relation extraction module: Calculates the distance between the vector representation of the query sentence and the prototype vectors of all relations, and extracts relations from the query sentence using the nearest neighbor concept.

[0049] This invention provides an electronic device, the device comprising:

[0050] At least one processor;

[0051] At least one GPU computing card; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor or the at least one GPU computing card to enable the at least one processor or the at least one GPU computing card to perform the method.

[0052] The present invention provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method described in any one of the embodiments.

[0053] The technology described in this application can complete the relation extraction task in incremental few-sample scenarios and improve the accuracy of relation extraction in such cases. Attached Figure Description

[0054] Figure 1 This is a flowchart of the present invention;

[0055] Figure 2 This is a structural diagram of the present invention;

[0056] Figure 3 A flowchart of a training basic relation extraction model provided by the present invention;

[0057] Figure 4 A flowchart for selecting representative samples for embedding is provided by the present invention;

[0058] Figure 5 A flowchart of a pseudo-incremental few-sample meta-task construction method provided by the present invention;

[0059] Figure 6 A flowchart illustrating the concept of integration relationship provided by this invention;

[0060] Figure 7 A flowchart for constructing a contrastive learning method is provided for this invention;

[0061] Figure 8 A flowchart of a training incremental relation extraction model provided by the present invention;

[0062] Figure 9 A flowchart for embedding fusion relationships is provided for this invention;

[0063] Figure 10 A structural diagram of an incremental few-sample relation extraction method provided by the present invention;

[0064] Figure 11 A schematic diagram of an incremental few-sample relationship extraction device provided by the present invention;

[0065] Figure 12 This is a block diagram of the electronic device of the present invention. Detailed Implementation

[0066] To enable those skilled in the art to better understand the present invention, the invention will be further described in detail below with reference to the accompanying drawings. This includes various details of the embodiments of this application to aid understanding, and these should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. The incremental few-sample relation extraction method for fusing relation concepts according to the present invention is performed according to the following steps:

[0067] a. Obtain a few-sample relation extraction dataset, and divide the dataset into a basic relation set, a new relation validation set, and a new relation test set to construct an incremental few-sample relation extraction dataset. The construction of the incremental few-sample relation extraction dataset is as follows: the entire dataset is divided into a basic relation set, a new relation validation set, and a new relation test set, wherein the three sets have no overlap and each contains a training sample set, a validation sample set, and a test sample set, and each sample set has no overlap. When incrementally learning new relations, a pseudo-incremental scenario is constructed from the sample set of the basic relations, which includes a support set and a query set. The former is used for model learning, and the latter is used for model validation.

[0068] b. Based on labeled basic relation samples, special markers are introduced, and sentence semantic information and entity pair semantic information are extracted using a pre-trained language model. The extraction of sentence-level semantics involves adding a special marker [CLS] at the beginning of the sentence and a special marker [SEP] at the end. The text is then context-encoded using a pre-trained BERT model, and the output at [CLS] is selected as the sentence-level semantic feature vector. The extraction of entity pair semantics involves adding special markers [unused0], [unused1], [unused2], and [unused3] at the beginning and end of the text. The processed text is input into the pre-trained BERT model, and the encoded representations of the beginning and end entity byte pairs are obtained using the auxiliary markers [unused0], [unused1], [unused2], and [unused3]. The average of these features is used as the entity pair semantic information, which is then concatenated with the sentence-level semantic information to obtain the final semantic feature vector containing both entity pairs and the sentence.

[0069] c. Train the prototype embedding of basic relations on labeled basic relation samples, train the basic relation extraction model based on the semantic feature vector of the training samples, and learn the prototype embedding representation of basic relations.

[0070] d. Fix the encoder and prototype embedding trained in the basic stage, and select representative samples of the basic relations based on the idea of ​​clustering; fixing the encoder and prototype embedding means fixing the learned basic relation prototype embedding representation and its pre-trained BERT model used for feature extraction, while retaining the parameter weights fine-tuned in this task; extracting representative samples of the basic relations means extracting the top N sample embedding representations most similar to the relation prototype embedding from the training samples of the basic relations based on the idea of ​​clustering, and fixing the average of the top N vector representations as the prototype embeddings of the basic relations in the incremental learning stage.

[0071] e. Construct a pseudo-incremental few-shot meta-learning task based on the basic relation set. Obtain the feature representation of relation concepts through a pre-trained MPNET model. Integrate the learning of new relations into the corresponding relation concepts. The construction of the pseudo-incremental few-shot meta-learning task is as follows: construct basic relations and new relations from the basic relation set, and construct a support set from the training set or test set of the basic relations. Integrate the support instances of new relations into the corresponding relation concepts as follows: the semantic feature vectors obtained by the fixed BERT model from a small number of support instances 1 or 5 samples of the new relation are used as text features. The vector representation of the relation concept is obtained through the pre-trained MPNET model. Then, the vector representation of the relation concept is fixed. Through contrastive learning, the embedding representation of the relation concept is integrated into the sentence-level semantic feature vector of the support instances. Finally, the vector representations belonging to the same relation are averaged to obtain the prototype representation of the new relation.

[0072] f. Train an incremental few-shot relation extraction model according to the construct meta-learning task; the training of the incremental few-shot relation extraction model is as follows: calculate the metric between the query sentence and the prototype through the prototype network, classify them, and train the incremental few-shot relation extraction model according to cross-entropy loss and contrastive loss.

[0073] g. Integrate the feature representation of the query sentence into relation prototypes, calculate the metric between the query sentence and the relation prototype vector, and extract the relation from the query sentence; the integration of the feature representation of the query sentence into relation prototypes is as follows: based on the similarity between the semantic feature vector of the query sentence and the prototype embeddings of the basic and newly added relations, assign similarity weights and generate a vector representation of the query sentence; the extraction of the relation from the query sentence is as follows: calculate the distance between the vector representation of the query sentence and all relation prototype vectors, and extract the relation from the query sentence using the nearest neighbor concept.

[0074] An incremental few-shot relation extraction device that integrates relation concepts is disclosed. This device comprises an incremental few-shot dataset construction module, a semantic encoder module, a basic relation extraction module, a relation concept injection module, a contrastive learning construction module, a query text feature generation module, and an incremental relation extraction module, wherein:

[0075] The incremental few-shot dataset construction module acquires the relation extraction dataset, processes the dataset, and constructs the incremental few-shot relation extraction dataset. Specifically, it includes a relation set splitting unit, used to divide the entire dataset into a base relation set and a new relation set, used to construct the incremental few-shot learning scenario; a dataset splitting unit, used to divide the entire dataset into a training set, a validation set, and a test set; and a pseudo-incremental meta-task construction unit, used to construct multiple different pseudo-incremental few-shot meta-learning tasks during training and testing.

[0076] The semantic encoder module encodes text into high-dimensional vector features. Specifically, it includes a semantic information extraction unit, which adds special markers [CLS] at the beginning of a sentence and [SEP] at the end, and obtains semantic representations through a pre-trained BERT model; a relation concept semantic extraction unit, which adds special markers [CLS] at the beginning of relation concepts, and obtains vector representations of relation concepts through a pre-trained MPNET model; and an entity pair semantic extraction unit, which adds special markers at the beginning and end of entities in a sentence, extracts the byte features of entity pairs, and calculates the average to obtain high-dimensional feature representations of the beginning and end entities.

[0077] Basic relation extraction module: Learns the prototype embedding representation of basic relations using the nearest neighbor concept by using labeled samples of the basic relation set.

[0078] Relationship concept injection module: For the newly obtained text representation of a relation, copy its sentence-level vector representation and pass it through a fully connected layer and activation function to construct a contrastive loss and incorporate its corresponding relation concept; to ensure the consistency of feature dimensions, copy the text representation of the basic relation and concatenate it with its sentence-level feature vector and the original semantic vector.

[0079] Contrastive learning construction module: Samples of newly added relations are constructed as positive examples, and samples of other new relations and basic relations are constructed as negative examples. The text feature vector representation that is easier to divide is obtained by contrastive loss.

[0080] Query text feature generation module: Based on the similarity between the semantic feature vector of the query sentence and the prototype embeddings of the basic and newly added relations, similarity weights are assigned to generate a vector representation of the query sentence;

[0081] Incremental relation extraction module: Calculates the distance between the vector representation of the query sentence and the prototype vectors of all relations, and extracts relations from the query sentence using the nearest neighbor concept.

[0082] An electronic device includes at least one processor; at least one GPU computing card; and at least one memory communicatively connected to the processor; wherein the memory stores instructions executable by the at least one processor or the at least one GPU computing card, such that the at least one processor or the at least one GPU computing card can perform the method.

[0083] A non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to cause the computer to perform the method described thereon;

[0084] Figure 1 The flowchart of this invention illustrates an incremental classification of new relationships in the case of a small number of samples. This method can be executed by an incremental few-sample device, which can be implemented in software and / or hardware. See also... Figure 1 The incremental few-sample relation extraction method for the fusion relation concept provided in this application includes:

[0085] S110. Obtain the relation extraction dataset, divide the dataset, and construct an incremental few-sample relation extraction dataset;

[0086] In one embodiment, the dataset partitioning specifically involves dividing the entire dataset into a basic relation set, a newly added validation relation set, and a newly added test relation set. Each relation set, to better reflect real-world scenarios, further divides the entire dataset into a basic relation set containing relationships, and a newly added validation relation set and a newly added test relation set with a small number of labeled samples. The three relation sets do not contain overlapping relation categories. A corresponding training set, validation set, and test set are then constructed based on each relation set.

[0087] For example, the relation extraction dataset can be the FewRel dataset, which consists of 80 relations, with 100 labeled samples for each relation; for relation set partitioning, the basic relation set can be 54 relations such as parent-child relations, which serve as the relation set with labeled samples; the newly added validation relation set can be 10 relations such as occupation relations; and the newly added test relation set can be 16 relations such as spouse relations.

[0088] S120. Based on the labeled basic relation samples, extract sentence semantic information and entity pair semantic information. The sentence-level semantic information and entity pair semantic information are both classification features of the basic relation and newly added relation extraction models.

[0089] In one embodiment, the extracted sentence-level semantic information can be understood as semantic information representing this text, which implies potential relationships; the extracted entity pair semantic information can be understood as contextual semantic information of entity pairs that have relationships in this text.

[0090] S130. Train the prototype embedding of the basic relation based on the labeled samples of the basic relation;

[0091] In one embodiment, the incremental learning scenario has a base model by default. New knowledge is learned incrementally based on the base model. Training the prototype embedding of the base relation can be understood as regular supervised learning, using the labeled samples of the base relation to learn its prototype embedding representation.

[0092] For example, the prototype embedding representation can be a high-dimensional feature vector;

[0093] S140. Fix the encoder and prototype embedding trained in the basic phase, and select representative samples of the basic relations.

[0094] In one embodiment, fixing the pre-trained model after fine-tuning in the base stage and the learned prototype embedding can be understood as retaining the model parameters learned in the base stage training. Secondly, saving the overall parameters trained in the base stage is costly. Selecting representative samples of the base relationship is to reduce the cost of retaining the trained base model.

[0095] For example, the trained encoder can be a BERT pre-trained model, and the representative sample can be the top 100 samples in the basic relation training set that are most similar to its relation prototype embedding.

[0096] S150. Construct multiple sets of different pseudo-incremental few-sample meta-learning tasks, and integrate the corresponding relation concepts into the support instances of newly added relations.

[0097] In one embodiment, constructing a multi-pseudo-incremental few-sample meta-learning task can be understood as utilizing existing samples of basic relations to construct a pseudo-incremental few-sample meta-learning task for model learning. In the incremental few-sample scenario, there are few labeled samples of newly added relations. The fusion of the corresponding relation concepts can be understood as fusing the semantic information of the relation concepts when extracting semantic features, thereby enhancing the feature representation supporting the instance.

[0098] For example, the newly added relationship can be a spousal relationship, and the relationship concept can be that the object of a subject is its spouse;

[0099] S160. Train the incremental few-sample relation extraction model according to the constructed meta-learning task.

[0100] In one embodiment, the model is trained to incrementally learn its ability through the constructed pseudo-incremental few-shot meta-task, and the trained model has the ability to incrementally learn new relationships in few-shot scenarios.

[0101] S170. Integrate the feature representation of the query sentence into the relation prototype, calculate the metric between the relation prototype vector and the query sentence to extract the relation;

[0102] In one embodiment, the method for calculating the metric can be Euclidean distance, etc., and the model classifies query sentences by calculating the metric between the query sample and all relation prototypes;

[0103] Figure 2 This embodiment provides a structural diagram of a feature extraction method. The extracted feature vector mainly consists of three parts: sentence-level semantic representation, semantic representation at the beginning of the sentence, and semantic representation at the end of the sentence. Contextual semantic features are extracted using a pre-trained BERT model. Specifically, a special marker [CLS] is added at the beginning of the sentence, and a special marker [SEP] is added at the end. Special markers [unused0], [unused1], [unused2], and [unused3] are added at the beginning and end of the text. The processed text is then context-encoded using the BERT model, and the output at [CLS] is used as the sentence-level semantic feature vector. The encoded representations of the beginning and end entity byte pairs are obtained using the auxiliary marker [unused*], and the average of these features is used as the semantic information of the entity pairs. The features of the two are combined. The final semantic feature vector, which includes entity pairs and sentences, is obtained.

[0104] For example, for extracting sentence and entity pair semantic information from the text, such as the input text "Zhang San was born in Xinjiang", the text after adding special tags is [CLS][unused0]Zhang San[unused1]born in [unused2]Xinjiang[unused3][SE P], and the output of [CLS] after the pre-trained BERT model is used as its sentence-level representation; based on [unused], the head entity Zhang San and the tail entity Xinjiang are located, and the average of the bytes corresponding to the head and tail entities after byte pair encoding is used as the semantic feature vector of the entity pair;

[0105] Figure 3 This invention provides a flowchart for training a basic relation extraction model. The model learns prototype embeddings based on training samples of basic relations. The training basic relation extraction model provided by this solution includes:

[0106] S310, using training samples from the basic relation set.

[0107] In one embodiment, the training samples of the basic relation set are the training sample set of the basic relation set when the dataset is partitioned.

[0108] S320, Feature Extraction Module.

[0109] In one embodiment, the feature extraction module structure participates Figure 2 The feature vectors of the text are extracted and used as the input to the model.

[0110] S330. Classify using the nearest neighbor concept and train using cross-entropy loss.

[0111] In one embodiment, training the basic relation extraction model specifically involves making the embedding representations of training samples of the same relation as similar as possible, learning the prototype embedding of the basic relation based on the training samples of the basic relation, initializing the prototype vector representation of the basic relation, and learning the prototype embedding representation of the basic relation based on the semantic feature vectors of its training samples and the prototype network.

[0112] Classification probability is The loss function is Where p is the probability of correctly extracting a relation, and Q is a test instance;

[0113] For example, for the basic parent-child relationship, such as the training text "Zhang San is Li Si's father" and "Wang Wu is Li Si's father", texts containing the same parent-child relationship will be included, according to the above. Figure 2 The extracted embedding representation is used to encode text embeddings with the same relationship as similar as possible using BERT, thereby training the prototype embedding b of the basic relationship. For test examples of the basic relationship, classification is performed based on the similarity between the encoded representation of the test sample and the prototype embedding of the basic relationship.

[0114] Figure 4 The present invention provides a flowchart for selecting representative sample embeddings, which retains the prototype of the learned basic relationship by selecting representative samples of the basic relationship, so as to retain the basic relationship learned by the model;

[0115] S410. Obtain the embedding representation of all training samples of the basic relation;

[0116] In one embodiment, for all training samples of the basic relation, according to the feature extraction model, see [link to relevant documentation]. Figure 2 Obtain the embedded feature vectors of all samples;

[0117] S420. Calculate the similarity between the prototype embedding of each relation and its training sample embedding representation;

[0118] In one embodiment, calculating the similarity between the embedding representation of all training samples and the prototype embedding representation learned by the base model can be understood as measuring the degree to which the training samples represent their relational embeddings.

[0119] For example, the method for calculating the metric can be a method for measuring correlation, such as Euclidean distance;

[0120] S430. Select the N most similar samples;

[0121] In one embodiment, retaining the top N samples that are most similar to the basic relation prototype embedding can preserve the relation embeddings learned by the basic model as much as possible during the incremental learning phase.

[0122] S440. Fix the mean representation of the N sample embeddings as the prototype embedding representation of the basic relationship in the incremental stage;

[0123] In one embodiment, the relationship is embedded as a feature. By averaging the features of the first N samples, the average value of the embedded representation of the N samples is fixed and used as the prototype embedded representation of the basic relationship in the incremental stage.

[0124] Figure 5 This is a flowchart illustrating a method for constructing a pseudo-incremental few-shot meta-learning task provided by the present invention. To train an incremental model, the pseudo-incremental few-shot meta-learning task must closely resemble the real-world scenario, enabling the model to acquire the ability to learn again.

[0125] S510. Construct the basic and newly added relations in the pseudo-incremental task from the basic relation set;

[0126] In one embodiment, B types of relations are randomly selected from the set of basic relations as basic relations, and a small number of N types of relations are randomly selected as new relations;

[0127] S520. Construct a support set from the training set of the newly constructed relations;

[0128] S530. Construct a query set from the verification set of the constructed basic relations and newly added relations;

[0129] In one embodiment, the constructed support set is used for model learning, and the query set is used for model validation and testing. For newly added relations, K samples are randomly selected from their corresponding training set to form the support set S. A query set Q is formed by randomly selecting q samples from the test set of all relations.

[0130] For example, in the FewRel dataset mentioned above, with 3 basic relationships and 2 new relationships, the test set contains query samples of 5 relationships. Therefore, a meta-task with 3 basic relationships and 2 new relationships needs to be constructed, i.e., B=3 and N=2. If the final expectation is to classify the new sentences under the condition that there are only 5 samples for each new relationship, then K=5. First, select 3 relationships from the 54 relationships in the training set of the basic relationship set as basic relationships, such as parent-child relationship, religious relationship, and participation relationship. Then select 2 relationships as new relationships, such as position relationship and membership relationship. From the two new relationships, select 5 sentences for each and 10 sentences for the other as the support set for the new relationships. Randomly select 5 samples from the validation set of each of the 5 relationships, for a total of 25 sentences. These sentences constitute the query set.

[0131] Figure 6 This invention provides a flowchart for incorporating relational concepts. When extracting semantic features from supporting instances of the newly added relation, the semantic information of the corresponding relational concept is fused to enhance the feature representation of the supporting instances.

[0132] S610. The relation concept is processed by MPNET to obtain the feature representation with semantic information;

[0133] In one embodiment, the relation concept is the descriptive text of this relation, and its sentence-level feature vector is obtained through a pre-trained model with semantic encoding capabilities, thereby capturing the semantic information of the relation concept;

[0134] For example, the pre-trained model with semantic encoding capability can be a pre-trained model with semantic encoding capability such as MPNET;

[0135] S620, Copy support instance, query instance sentence-level feature vector;

[0136] S630. Obtain the transformed embedding representation through a fully connected layer and an activation function;

[0137] In one embodiment, there is a discrepancy between the content and structure of the relation description and the supporting instance text, which is addressed by a fully connected layer.

[0138] S640. Set the contrastive loss to integrate the relational concept feature representation into the sentence-level feature vector representation that supports instances;

[0139] For example: the relation concept is pre-trained using MPNET to obtain the semantically informational relation concept feature representation, denoted as C; to ensure feature consistency, the sentence-level feature vectors of the supporting instances and query instances are copied, denoted as S; since there are discrepancies in the content and structure of the relation description and the supporting instance text, a fully connected layer is used to resolve the discrepancy problem. The sentence-level representation of the supporting instances passes through the fully connected layer, i.e., ... The concept of relationship is incorporated through the design of contrastive loss, where the loss is... in, f is the function for calculating Euclidean distance.

[0140] Figure 7 The present invention provides a flowchart for constructing a contrastive learning process, which constructs positive and negative examples of basic relations and newly added relations, and the positive and negative examples are further used to obtain the prototype embedding representation of the newly added relations after the fusion of relation concepts;

[0141] S710. Construct a pseudo-new relation support set from the training set of basic relations; construct a pseudo-new relation query set from the test set of basic relations.

[0142] In one embodiment, the model is trained using the constructed support set and validated using the test set;

[0143] S720. Construct positive and negative sample cases for different relations;

[0144] For example, positive and negative examples are constructed for both basic and newly added relationships. Positive and negative examples are constructed for the basic class and the newly added class. Samples belonging to its relationship are positive examples, and the rest are negative examples.

[0145] S730, Set the contrast loss to bring the prototype closer to its positive examples and widen the distance between it and the negative examples;

[0146] In one embodiment, positive and negative examples are introduced to help the incremental model learn a prototype representation that better fits the newly added relationship. For the constructed positive and negative examples, samples belonging to the relationship are considered positive examples, and the rest are considered negative examples. The prototype embedding of the new relationship is constrained by constructing a triplet loss, the specific loss of which is as follows.

[0147]

[0148] Figure 8 This invention provides a flowchart for training an incremental relation extraction model. The incremental few-shot relation extraction model is trained according to the constructed meta-learning task, where the prototype of the basic relation is obtained by averaging the feature embeddings of its representative samples. To enable the model to better learn the prototype features of newly added relations in few-shot scenarios, this application implements the integration of relation concepts to enhance the learning of the prototype embeddings of newly added relations.

[0149] S810. Obtain the prototype embedding representation of the basic relation based on the representative samples of the basic relation;

[0150] In one embodiment, the acquisition of the basic relation prototype embedding is represented by the mean of the embeddings of its representative samples;

[0151] S820. Learn the prototype embedding representation of the newly added relation through the feature vectors of the support examples after incorporating the relation concept using a pseudo-incremental task. The process of incorporating the relation concept can be found in [link to documentation]. Figure 6 ;

[0152] S830. The feature representation of the query sentence is integrated into the relation prototype to obtain the enhanced feature representation;

[0153] In one embodiment, the process of incorporating a query sentence into a relational prototype can be found in [reference needed]. Figure 9 The prototype embeddings of basic relations and newly added relations are respectively merged to obtain the enhanced text encoding representation;

[0154] S840. Classify the query sentence based on the measure between the query sentence and the prototype through the prototype network, and train the incremental model based on cross-entropy loss and contrastive loss.

[0155] In one embodiment, the constructed meta-learning task and the constructed positive and negative examples are used to train an incremental few-shot relation extraction model based on the prototype network. Classification is performed based on the similarity between the query sentence encoding and the relation prototype embedding, with classification probabilities...

[0156] for The final model is trained on the constructed pseudo-incremental learning task, using cross-entropy loss to train the incremental few-sample relation extraction model, with the loss being...

[0157] Figure 9 The present invention provides a structural diagram of fused relationship embedding. For the embedding representation of query text, the prototype embeddings of basic relationships and the prototype embeddings of newly added relationships are fused respectively to calculate the similarity with the prototype embeddings of all relationships.

[0158] S910. Embedded representation of the query sentence. The embedded representation of the query text is obtained according to the feature extraction method described above;

[0159] S920, Calculate similarity;

[0160] In one embodiment, for the relation prototype (P) bass P novel ) and high-dimensional feature representation of query sentences (X q The similarity between the query sentences is analyzed, and similarity weights are assigned to generate vector representations of the query sentences. The similarity weights are calculated as follows: w n =1-w b The final vector representation of the query sentence incorporating the underlying relations is as follows: The vector representation incorporating the newly added relation is as follows:

[0161] For example, the method for calculating the metric can be a method for measuring correlation, such as Euclidean distance;

[0162] Figure 10 This is a structural diagram of an incremental few-shot relation extraction method provided in an embodiment of this application. The incremental few-shot relation model that integrates relation concepts is mainly divided into two parts: a basic relation model and an incremental relation model. The basic relation model is described in [link to relevant documentation]. Figure 10-10 The gray areas in section 10;

[0163] In one embodiment, the overall process of model training can be understood as first using labeled samples of basic relations to train a basic relation extraction model; second, when there are only a few labeled samples of new relations, the model incrementally learns new relations in a meta-learning manner.

[0164] Figure 11 This is a schematic diagram of the structure of incremental few-shot relation extraction that integrates relation concepts according to the present invention, including: an incremental few-shot dataset construction module, a semantic encoder module, a basic relation extraction module, a relation concept injection module, a contrastive learning construction module, a query text feature generation module, and an incremental relation extraction module;

[0165] This invention provides an electronic device and a readable storage medium;

[0166] like Figure 12 The diagram shown is a block diagram of the electronic device of the present invention. The electronic device refers to various modern electronic digital computers, including personal computers, portable computers, and various server devices. The components, their interconnections, and functions shown herein are merely examples.

[0167] like Figure 12 As shown, the electronic device includes: one or more multi-core processors, one or more GPU computing cards, and memory. To enable interaction between the electronic device, it should also include: input devices and output devices; the various devices are interconnected and communicate with each other via a bus.

[0168] The memory is the non-transitory computer-readable storage medium provided in this application, wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor or the at least one GPU computing card to enable the at least one processor or the at least one GPU computing card to perform the method described in any one of the embodiments of this application;

[0169] Input devices provide and accept control signals that users input into electronic devices, including keyboards that generate numerical or character information and mice used to control the devices to generate other key signals; output devices provide feedback information to user electronic devices, including displays that print execution results or processes.

[0170] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practical disclosure. This application is intended to cover any variations, uses, or adaptations of the invention, which should be understood to be in accordance with the fact that the invention is not limited to the precise structures described above and shown in the drawings, and that various modifications and changes can be made without departing from its scope.

Claims

1. An incremental few-sample relation extraction method for fusing relation concepts, characterized in that... Follow these steps: a. Obtain a few-sample relation extraction dataset, divide the dataset into a basic relation set, a new relation validation set, and a new relation test set, and construct an incremental few-sample relation extraction dataset; b. Based on labeled basic relation samples, special markers are introduced, and pre-trained language models are used to extract sentence semantic information and entity pair semantic information; c. Train the prototype embedding of basic relations on labeled basic relation samples, train the basic relation extraction model based on the semantic feature vector of the training samples, and learn the prototype embedding representation of basic relations. d. After fixing the encoder and prototype embedding trained in the basic stage, select representative samples of basic relationships based on the idea of ​​clustering; e. Construct a pseudo-incremental few-shot meta-learning task based on the basic relation set. Obtain the feature representation of relation concepts through a pre-trained MPNET model. Integrate the corresponding relation concepts into the learning of new relations. The construction of the pseudo-incremental few-shot meta-learning task is as follows: construct basic relations and new relations from the basic relation set, and construct a support set from the training set or test set of the basic relations. Integrate the corresponding relation concepts into the support instances of new relations. The semantic feature vectors obtained by the fixed BERT model from a small number of support instances 1 or 5 samples of the new relation are used as text features. The vector representation of the relation concept is obtained through the pre-trained MPNET model. Then, the vector representation of the relation concept is fixed. Through contrastive learning, the embedding representation of the relation concept is integrated into the sentence-level semantic feature vector of the support instances. Finally, the vector representations belonging to the same relation are averaged to obtain the prototype representation of the new relation. f. Train the incremental few-sample relation extraction model based on the constructed meta-learning task; g. Integrate the feature representation of the query sentence into the relation prototype, calculate the metric between the relation prototype vector and the feature representation, and extract the relation from the query sentence.

2. The incremental few-sample relation extraction method for fusion relation concepts according to claim 1, characterized in that, The construction of the incremental few-sample relation extraction dataset mentioned in step a is as follows: the entire dataset is divided into a basic relation set, a new relation validation set, and a new relation test set. The three sets have no overlap and each contains a training sample set, a validation sample set, and a test sample set. Each sample set has no overlap.

3. The incremental few-sample relation extraction method for fusion relation concepts according to claim 1, characterized in that, The extraction of sentence-level semantics in step b involves adding a special marker [CLS] at the beginning of the sentence and a special marker [SEP] at the end. The text is then context-encoded using a pre-trained BERT model, and the output at [CLS] is selected as the sentence-level semantic feature vector. The extraction of entity-pair semantics involves adding special markers [unused0], [unused1], [unused2], and [unused3] to the beginning and end entities in the text. The processed text is then input into the pre-trained BERT model, and the encoded representations of the beginning and end entity byte pairs are obtained using the auxiliary markers [unused0], [unused1], [unused2], and [unused3]. The average of its features is used as the semantic information of the entity pair, which is then concatenated with the sentence-level semantic information to obtain the final semantic feature vector that includes both entity pairs and sentences.

4. The incremental few-sample relation extraction method for fusion relation concepts according to claim 1, characterized in that, The fixed encoder and prototype embedding mentioned in step d are as follows: the learned basic relation prototype embedding representation and its pre-trained BERT model used for feature extraction are fixed, and the parameter weights finely tuned in this task are retained; the representative samples of the basic relation are extracted as follows: based on the idea of ​​clustering, the top N sample embedding representations that are most similar to the relation prototype embedding are extracted from the training samples of the basic relation, and the average top N vector representations are fixed and used as the prototype embeddings of the basic relation in the incremental learning stage.

5. The incremental few-sample relation extraction method for fusion relation concepts according to claim 1, characterized in that, The training of the incremental few-shot relation extraction model in step f is as follows: the query sentence is compared with the prototype by calculating the metric between the query sentence and the prototype through the prototype network, and then classified. The incremental few-shot relation extraction model is trained based on the cross-entropy loss and the contrastive loss.

6. The incremental few-sample relation extraction method for fusion relation concepts according to claim 1, characterized in that, The feature representation of the query sentence in step g is integrated into the relation prototype by: assigning similarity weights based on the similarity between the semantic feature vector of the query sentence and the prototype embeddings of the basic and newly added relations, and generating a vector representation of the query sentence; the relation extraction of the query sentence is: calculating the distance between the vector representation of the query sentence and all relation prototype vectors, and using the nearest neighbor idea to extract the relation of the query sentence.

7. An incremental few-sample relation extraction device for fusion relation concepts, used to implement the incremental few-sample relation extraction method for fusion relation concepts as described in claim 1, characterized in that, The device consists of an incremental few-shot dataset construction module, a semantic encoder module, a basic relation extraction module, a relation concept injection module, a contrastive learning construction module, a query text feature generation module, and an incremental relation extraction module, wherein: The incremental few-shot dataset construction module acquires the relation extraction dataset, processes the dataset, and constructs the incremental few-shot relation extraction dataset. Specifically, it includes a relation set splitting unit, used to divide the entire dataset into a base relation set and a new relation set, used to construct the incremental few-shot learning scenario; a dataset splitting unit, used to divide the entire dataset into a training set, a validation set, and a test set; and a pseudo-incremental meta-task construction unit, used to construct multiple different pseudo-incremental few-shot meta-learning tasks during training and testing. The semantic encoder module encodes text into high-dimensional vector features. Specifically, it includes a semantic information extraction unit, which adds special markers [CLS] at the beginning of a sentence and [SEP] at the end, and obtains semantic representations through a pre-trained BERT model; a relation concept semantic extraction unit, which adds special markers [CLS] at the beginning of relation concepts, and obtains vector representations of relation concepts through a pre-trained MPNET model; and an entity pair semantic extraction unit, which adds special markers at the beginning and end of entities in a sentence, extracts the byte features of entity pairs, and calculates the average to obtain high-dimensional feature representations of the beginning and end entities. Basic relation extraction module: Learns the prototype embedding representation of basic relations using the nearest neighbor concept through labeled samples of the basic relation set; Relationship concept injection module: For the newly obtained text representation of a relation, copy its sentence-level vector representation and pass it through a fully connected layer and activation function to construct a contrastive loss and incorporate its corresponding relation concept; to ensure the consistency of feature dimensions, copy the text representation of the basic relation and concatenate it with its sentence-level feature vector and the original semantic vector. Contrastive learning construction module: Samples of newly added relations are constructed as positive examples, and samples of other new relations and basic relations are constructed as negative examples. The text feature vector representation that is easier to divide is obtained by contrastive loss. Query text feature generation module: Based on the similarity between the semantic feature vector of the query sentence and the prototype embeddings of the basic and newly added relations, similarity weights are assigned to generate a vector representation of the query sentence; Incremental relation extraction module: Calculates the distance between the vector representation of the query sentence and the prototype vectors of all relations, and extracts relations from the query sentence using the nearest neighbor concept.

8. An electronic device, characterized in that, The device includes at least one processor; at least one GPU computing card; and at least one memory communicatively connected to the processor; wherein the memory stores instructions executable by the at least one processor or the at least one GPU computing card, such that the at least one processor or the at least one GPU computing card can perform the method of any one of claims 1-6.

9. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.

Images