Class increment knowledge acquisition method based on prototype network gating mechanism

Abstract

The invention provides a class increment knowledge acquisition method based on a prototype network gating mechanism. The method comprises the following steps: acquiring a to-be-predicted text; inputting the text into a coding module obtained by training through a class incremental learning method of a knowledge acquisition field to obtain a feature vector of each lexical element, and constructing a unit representation of each knowledge unit by the feature vector of each lexical element; determining the category to which each knowledge unit belongs by utilizing a preset prototype network gating mechanism, namely obtaining the similarity between the unit representation of each knowledge unit and the prototype representation of each category, and screening partially matched expert networks according to the similarity to predict the knowledge units, obtaining a prediction value of whether the screened expert network pair knowledge unit belongs to the category corresponding to the expert network; for the overlapped knowledge units, only the predicted value of the knowledge unit with the highest predicted value is reserved; and if the predicted values of one knowledge unit in all categories are smaller than a preset threshold value, the knowledge unit does not belong to the category corresponding to any expert network.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, specifically to the field of incremental learning, and more specifically to a quasi-incremental knowledge acquisition method based on a prototype network gating mechanism. Background Art

[0002] As the Internet plays an increasingly important role in our daily lives, the massive amount of unstructured text data has brought great challenges to people's access to information. How to help people quickly acquire useful knowledge has become an urgent problem to be solved. In this context, information extraction technology has emerged and has become one of the key tasks in natural language processing technology. Information extraction technology is mainly used to extract useful knowledge, such as events and entities, from unstructured texts in order to build knowledge graphs, perform information retrieval, and assist other natural language processing tasks. As one of the important research directions in information extraction technology, event extraction mainly studies how to extract events of interest to users from unstructured texts and present them in a structured manner.

[0003] Named Entity Recognition (NER) and Relation Extraction (RE) tasks occupy a core position in the field of knowledge acquisition. They are crucial for understanding and extracting key information from text. However, when traditional knowledge acquisition models are faced with incremental learning of new categories of data, people hope that the model can gradually adapt to the new categories of data without losing the prediction performance of the old categories. But in reality, the model is prone to lose the memory of old knowledge while learning new knowledge; this phenomenon is particularly evident in quasi-incremental learning environments. To address this challenge, the model must not only be able to flexibly adapt to new data, but also maintain a stable memory of the learned knowledge during the continuous learning process.

[0004] The incremental learning of NER tasks is more complex than that of RE tasks. The challenges faced by incremental learning of NER tasks mainly stem from the particularity of the tasks, especially when dealing with backward incompatibility and forward incompatibility. The following takes the NER task as an example to discuss these two incompatibilities and their impact on the NER task:

[0005] 1) Forward incompatibility:

[0006] In the quasi-incremental learning scenario of the NER task, when new entity types are introduced for learning, the references of previously learned entity types may still appear in the training samples of the current task, but these samples do not provide corresponding annotations for these references. In this case, the model may forget the previously learned entity types or fail to correctly identify these unlabeled entity references because they do not appear as a focus in the training data of the current task. This leads to the so-called catastrophic forgetting problem, that is, the model loses or confuses old knowledge when learning new knowledge.

[0007] 2) Backwards Incompatibility:

[0008] In the current learning task, some non-entity references may be identified as a certain entity type in future tasks. This means that in the training phase of the current task, these references are regarded as non-entities, but in future learning tasks, they need to be identified and correctly classified. In this case, the model needs to be able to adapt to new entity types and re-evaluate and identify these references that were previously regarded as non-entities in future learning tasks. The present invention calls this phenomenon "belief fixation". In psychology, belief fixation means that once a certain idea is formed, it is difficult to change the original belief even in the face of evidence that directly refutes the idea. Even if the wrong idea is corrected, people will still be affected by the original wrong information, because the original wrong information has formed an impression in their cognitive structure, and changing this impression requires more cognitive effort than accepting new information.

[0009] Existing research on incremental learning in the field of knowledge acquisition mainly focuses on backward incompatibility. Knowledge distillation is usually adopted to use the previously learned model as the teacher model, predict the distilled pseudo-labels on the current sample, and then jointly learn the current student model through these labels and the current true labels, while ignoring the problem of belief fixation, resulting in the accuracy of knowledge acquisition needs to be improved.

[0010] It can be seen that in the field of knowledge acquisition, the existing technology has not effectively solved the problems of forward incompatibility and backward incompatibility.

[0011] It should be noted that this background technology is only used to introduce the relevant information of the present invention to help understand the technical solution of the present invention, but it does not mean that the relevant information is necessarily the prior art. The relevant information is submitted and disclosed together with the present invention solution. If there is no evidence that the relevant information has been disclosed before the application date of the present invention, the relevant information shall not be regarded as the prior art. Summary of the invention

[0012] Therefore, the purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a quasi-incremental knowledge acquisition method based on the prototype network gating mechanism.

[0013] The objective of the present invention is achieved through the following technical solutions:

[0014] According to a first aspect of the present invention, a class incremental learning method is provided, which is executed each time a new category needs to be recognized in a knowledge acquisition task, comprising: S1, creating a dedicated expert network for the newly added category in the model, the model comprising a BERT-based encoding module for extracting feature vectors of each word unit from an input text and an expert network for each category; the expert network for each category is used to extract category embeddings from unit representations of knowledge units in the input text, and to determine a prediction value of whether the knowledge unit belongs to the category based on the category embeddings, wherein the unit representation includes feature vectors of word units involved in the knowledge unit; S2, performing training in the preview stage, comprising: obtaining a dedicated training set in S1 for training whether the knowledge unit belongs to the newly added category, and using the dedicated training set and the binary intersection The cross entropy loss function is used to train the newly constructed expert network and encoding module to obtain the previewed encoding network and expert network; S3, using the latest encoding network to extract feature vectors for each word in the text of the proprietary training set of each category, to construct the unit representation of the knowledge unit, and clustering the unit representation of the knowledge unit of each category to obtain multiple clusters, and determining the representative prototype representation and extracted unit representation from each cluster, and using the original sample corresponding to the extracted unit representation to construct the core set of the category; S4, executing the review stage training, including: obtaining a mixed training set consisting of the proprietary training set of the newly added category and the core set of each category, and using the preset total loss function, prototype representation and mixed training set training to update the parameters of each expert network and encoding module.

[0015] Optionally, for each category, the method of determining a representative prototype representation and an extracted unit representation from each cluster includes: obtaining the unit representation corresponding to each recognition unit belonging to the category; clustering the unit representations of the category separately using a preset clustering algorithm to obtain multiple clusters under the category; using the centroid of each cluster under the category as the prototype representation of the cluster to obtain multiple prototype representations under the category; and sampling in equal proportion from each cluster under the category to obtain an extracted unit representation.

[0016] Optionally, the binary cross entropy loss function for the preview phase is:

[0017]

[0018] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p(k|s ij) represents the k-th expert network for the recognition unit s ij Output prediction value, prediction value in the preview phase Represents knowledge units ij , sigmoid(·) represents the sigmoid function.

[0019] Optionally, during the review phase, for the newly created expert network, whose input text comes from a proprietary training set, the preset total loss function includes a binary cross entropy loss function used to guide the learning of the newly created expert network:

[0020]

[0021] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p′(k|s ij ) represents the kth expert network set in the review phase to identify unit s ij The predicted value of the output, where:

[0022]

[0023]

[0024] r ij =W6·ReLU)W5·e ij +b1)+b2

[0025] r k =W6·ReLU(W5·p k +b1)+b2

[0026] Where sigmoid(·) represents the sigmoid function, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection, r k Indicates the use of projection network to represent the prototype p k The vector obtained by projection, W5 represents the weight parameter of the first layer of the projection network, b1 represents the bias parameter of the first layer of the projection network, W6 represents the weight parameter of the second layer of the projection network, b2 represents the bias parameter of the second layer of the projection network, ReLU represents the ReLU activation function, Indicates that ik The prototypes of the same category and closest distance represent: Indicates eik The shortest distance to the prototype representation of the same category, Indicates e ik is the shortest distance between the prototype representation of category l, exp(·) represents the exponential function, and K represents the total number of categories.

[0027] Optionally, for the expert network of non-new categories, the input text comes from the core set corresponding to the expert network, and the distillation loss function in the review stage is:

[0028]

[0029] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, n represents the total number of words in the input text, and K represents the total number of categories. Indicates the soft distillation tag that needs to be imitated. is the predicted value output by the model after the previous class incremental learning, p(k|s ij ) represents the kth expert network corresponding to the non-new category for the recognition unit s in the review stage ij The predicted value of the output, Optionally, the total loss function includes a weighted sum of a binary cross entropy loss function, a distillation loss function, and a contrast loss function, and the parameters of each expert network, encoding network, and projection network are optimized according to the total loss function to reduce the total loss, wherein the contrast loss function is:

[0030]

[0031] Where N represents the total number of recognition units, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection is Indicates that ij The prototypes of the same category and closest distance represent: Indicates category l and e ik The nearest prototype represents,τ1 represents the preset temperature hyperparameter.

[0032] Optionally, the knowledge acquisition task is a named entity recognition task or a relationship extraction task, wherein the recognition units under the named entity recognition task are the head word and the tail word of a paragraph in the input text, and the recognition units under the relationship extraction task are the word-grams corresponding to entities of two relationship categories to be identified in the input text.

[0033] According to a second aspect of the present invention, there is provided a class incremental knowledge acquisition method based on a prototype network gating mechanism, comprising: obtaining a text to be predicted; inputting the text to be predicted into an encoding module trained according to the method of the first aspect to obtain a feature vector of each word unit, and using the feature vector of each word unit to construct a unit representation of each knowledge unit; using a preset prototype network gating mechanism to determine the category to which each knowledge unit belongs, comprising: obtaining the similarity between the unit representation of each knowledge unit and the prototype representation of each category, selecting partially matched expert networks according to the similarity to predict the knowledge unit, and obtaining a prediction value of the selected expert network on whether the knowledge unit belongs to the category corresponding to the expert network; for overlapping knowledge units, only retaining the prediction value of the knowledge unit with the highest prediction value, and when the highest prediction value is greater than or equal to a preset threshold, judging that the retained knowledge unit belongs to the category corresponding to the expert network that outputs the highest prediction value; if the prediction values ​​of a knowledge unit in all categories are less than the preset threshold, the knowledge unit does not belong to the category corresponding to any expert network.

[0034] According to a third aspect of the present invention, there is provided an electronic device comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect and / or the second aspect by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0036] Figure 1 A schematic diagram of a flow chart of a quasi-incremental learning method according to an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of samples in a dedicated training set and samples in a regular training set according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] As mentioned in the background technology section, in the field of knowledge acquisition, the existing technology does not have a good solution to the problems of forward incompatibility and backward incompatibility. For the problem of forward incompatibility, the method of the present invention is based on the encoding module of BERT to capture the dependencies between each word in the input text; then, a number of prototype representations and core sets are constructed for each category, and these prototype representations and core sets can be continuously updated and optimized in each incremental learning to avoid the model losing its memory of old knowledge when learning new knowledge, and improve the prediction performance of old categories; secondly, for the problem of backward incompatibility, the method of the present invention implements a multi-expert hybrid prediction method by constructing an exclusive expert network for each newly added category. Each expert network can focus on learning the relevant knowledge of its corresponding category, improve the accuracy of predicting new category data, and solve the problem of backward incompatibility; thirdly, in the preview stage, the expert network is first allowed to focus on learning the knowledge of its own category in the proprietary training set, and then in the review stage, the knowledge of the newly added type of proprietary training set, prototype representation and core set is used to improve the prediction performance of each expert network.

[0040] The knowledge acquisition tasks involved in the method of the present invention include named entity recognition tasks and relationship extraction tasks. The following implementation mode 1 has been introduced by taking the implementation scenario involved in the named entity recognition task as an example. Implementation mode 2 introduces the implementation scenario involved in the relationship extraction task, but in order to avoid redundancy, it focuses on the differences relative to implementation mode 1.

[0041] Implementation Method 1

[0042] In the named entity recognition task, the goal of the task is to perform entity category recognition. It is necessary to identify whether each component element in the input text (such as characters, words, and phrases, and the division unit of the component elements depends on the type of natural language) is a non-entity category or a certain entity category.

[0043] For example: Assume that the categories that the task needs to identify include:

[0044] O: represents an ordinary word or phrase, not belonging to any named entity.

[0045] PER: stands for Person, which is used to mark the names of people or noun phrases related to people in the text.

[0046] ORG: stands for Organization, and is used to mark the organization name in the text, such as company, institution, group, etc.

[0047] LOC: stands for Location, and is used to mark geographical location names in text, such as cities, countries, rivers, mountains, etc.

[0048] MISC: stands for Miscellaneous category, which is used to mark other types of named entities that do not belong to the above PER, ORG, and LOC categories, such as work names, event names, proper nouns, etc.

[0049] Among them, O is a non-entity category, and PER, ORG, LOC, and MISC are entity categories.

[0050] Of course, there may be other forms of named entity recognition tasks according to the needs of the implementer. For example, another task is to mark each word in each sentence of the input text as B, I, E, S, and O. "O" means that the corresponding word is not an entity (non-entity category), "S" means that the corresponding word is a separate entity, and "B", "I", and "E" respectively indicate that the corresponding word is the beginning, middle, and end of an entity.

[0051] According to one embodiment of the present invention, see Figure 1 , a class incremental learning method, which is executed each time a new recognized category is required in a knowledge acquisition task, and comprises steps: A1, A2, A3, and A4. In order to better understand the present invention, each step is described in detail below in conjunction with a specific embodiment.

[0052] Step A1: Create a dedicated expert network for the newly added category in the model, the model includes a BERT-based encoding module for extracting feature vectors of each word from the input text and an expert network for each category; the expert network for each category is used to extract category embeddings from the unit representation of the knowledge unit in the input text, and determine the prediction value of the knowledge unit belonging to the category based on the category embedding, wherein the unit representation includes the feature vectors of the words involved in the knowledge unit.

[0053] For the existing named entity recognition task, after using the BERT model, it is necessary to add the conditional random field (CRF) and the Viterbi algorithm to complete the named entity recognition, which is equivalent to asking an expert to complete the labeling of all categories. Therefore, it is easy to cause the problem of forward and backward incompatibility. Therefore, the embodiment of the present invention makes improvements on this.

[0054] According to one embodiment of the present invention, the model is a named entity recognition model. When initially constructing the named entity recognition model, X categories of named entity recognition known to the implementer at that time can be used as the named entity recognition model to add X expert networks after the encoding module. The encoding module can use the BERT model; the expert network can be constructed using a neural network, such as a feedforward neural network. Then, the BERT model and the X expert networks are trained based on a training set with relevant annotated labels. Each expert network is dedicated to the recognition of one category. If the implementer needs to add a new category, a new exclusive expert network is created for each new category in addition to the X expert networks.

[0055] The encoding module uses the BERT model as a context encoder to capture the dependencies between the various components in the input text. n ] to represent the input text sequence X, x1~x n The n components of encoding 1 to n (corresponding to n word tokens) are respectively sent to the BERT encoder to obtain the contextual hidden vector H corresponding to the input text. The process can be expressed as:

[0056] H=BERT(X)

[0057] in, Among them, h1~h n The feature vectors of the n components corresponding to the codes 1 to n, d j Represents the dimension of the feature vector.

[0058] According to an embodiment of the present invention, the knowledge unit is set as a paragraph in a sentence, which can be represented as s ij , that is, a paragraph starting with word number i and ending with word number j. Each paragraph may contain one component element (i=j), or may contain multiple components (two or more, i<j). In the embodiment of the present invention, if a knowledge unit is finally identified as a certain category, it means that the knowledge unit (paragraph s ij ) is identified as the category. The unit representation of the knowledge unit is set as the feature vector of the token numbered i and the feature vector of the token numbered j: (h i ,h j ), where h i represents the feature vector of the word number i extracted by the encoding module, h j Represents the feature vector of the word number j extracted by the encoding module.

[0059] According to one embodiment of the present invention, the expert network is configured to use two feedforward neural networks (FFNs) with different feedforwards to process the feature vectors of two word units in the unit representation of the knowledge unit to extract the category embedding, which is expressed as:

[0060]

[0061] in, Represents the expert network of category k for the text segment s ij The extracted category embedding, s, e, k represent the start position, end position, and kth entity category, respectively, FFN s,k Indicates the text segment ij The feedforward neural network, FFN, processes the feature vector of the word number i in e,k Indicates the text segment ij The feedforward neural network that processes the feature vector of the word number j in d o FFN s,k and FFN e,k The output dimension of k is . Modeling the clear separation of each entity category in each paragraph can promote learning and knowledge distillation, and reduce the interference between multiple tasks. Organize into a matrix The upper triangular area of ​​​​the The row and column indicate the start and end respectively.

[0062] Therefore, the feed-forward neural network FFN of the k-th entity category can be s,k and FFN e,k As the "experts" of this category, the "experts" provide the model with deep insights into the recognition of a certain category. Through the integration of these experts, the method of the present invention can provide customized processing strategies for different entity categories.

[0063] Step A2: Execute the training in the preview stage, including: obtaining the proprietary training set in S1 for training whether the knowledge unit belongs to the newly added category, using the proprietary training set and the binary cross entropy loss function to train the newly created expert network and encoding module to obtain the previewed encoding network and expert network.

[0064] According to an embodiment of the present invention, a dedicated training set may include multiple samples, each sample is a sentence or a paragraph of text; the label information in the dedicated training set marks whether each component element in each sample belongs to the category to be recognized by the corresponding expert network. For example, for a certain component element, if the corresponding value in the label information is 1, it means it belongs to the category, and if it is 0, it means it does not belong to the category. The input text can be a sample; or, it can be spliced ​​from multiple samples as the input text. It should be noted that in conventional named entity recognition datasets, different categories are marked on the samples at the same time. See Figure 2 , which shows the difference between a proprietary training set and a regular data set; in a regular data set, for a sample, its label information contains all possible category labels. For example: for the sample "Emma at NASAlaunched asatellite", the true value indicated by the label T is: "PER O ORG OOO", indicating that the first element in the sample is a person's name, the third element is an organization name, and the remaining elements are all non-entity types. After changing to a proprietary training set of a certain category, only the elements belonging to that category will be labeled as that category (indicated by 1) in the proprietary training set, and other categories can be labeled as non-entity categories (or other categories, indicated by 0), such as shown in labels T1 and T2.

[0065] In addition, when it is desired to simulate the method of the present invention with an existing data set, a conventional data set, such as the CoNLL-2003 data set or the Ontonotes-5.0 data set, can be used for simulation. For example: First, according to the conventional data set, prepare the proprietary training sets for each part; then select some categories from all the entity categories contained in the data set and train a model first; then, each time select a category from other unselected categories as a new category, implement the method of the present invention to observe the effect. However, in actual scenarios, the implementer can train a model based on the proprietary training set of all categories currently known to him, and then, if he wants to add a new category, he can label samples for the new category and construct a proprietary training set; and create a new expert network and train it according to the method of the present invention.

[0066] According to one embodiment of the present invention, in the preview stage, only classification loss is used to supervise the training of the model. Since the expert network of each network is exclusive, each expert network is a binary classification network, and a binary classification cross entropy loss function is used for simplicity. Preferably, the binary classification cross entropy loss function in the preview stage is:

[0067]

[0068] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p(k|s ij ) represents the k-th expert network for the recognition unit s ij Output prediction value, prediction value in the preview phase Represents knowledge units ij , sigmoid(·) represents the sigmoid function. In the pre-training phase, the way to update parameters can be customized by the implementer. For example, the implementer can set a first learning rate for the newly created expert network and a second learning rate for the encoding module as needed, and the second learning rate is much smaller than the first learning rate to fine-tune the encoding module during pre-training. Alternatively, if the implementer wants to change only the parameters of the newly created expert network, the parameters of the encoding module can be frozen.

[0069] Step A3: Using the latest encoding network, extract feature vectors from each word in the text of the proprietary training set of each category to construct the unit representation of the knowledge unit, and cluster the unit representation of the knowledge unit of each category to obtain multiple clusters, determine the representative prototype representation and extracted unit representation from each cluster, and use the original samples corresponding to the extracted unit representation to construct the core set of the category;

[0070] Although the previous steps set up a dedicated expert network for each category. However, when faced with the dynamic increase of categories, how to effectively route and integrate the predictions of these experts becomes a challenge. If a neural network routing module with a fixed output length is used, it will face the problem of difficulty in expansion. If the routing module is retrained at each step of incremental learning, new catastrophic forgetting problems will inevitably be introduced in the routing module.

[0071] According to one embodiment of the present invention, for each category, the method of determining a representative prototype representation and an extracted unit representation from each cluster includes: obtaining the unit representation corresponding to each recognition unit belonging to the category; clustering the unit representations of the category using a preset clustering algorithm to obtain multiple clusters under the category; using the centroid of each cluster in the multiple clusters under the category as the prototype representation of the cluster to obtain multiple prototype representations under the category; sampling in equal proportion from each cluster under the category to obtain the extracted unit representation. The determination of each recognition unit belonging to a certain category can be carried out by extracting the text belonging to the category in the dedicated training set with the help of annotated labels, and then determining the unit representation of this part of the text. Subsequently, based on the text, the text belonging to the category is clustered using a preset clustering algorithm (such as the K-means algorithm).

[0072] Indicatively, after pre-training, the trained model is M k , using the latest model to extract the feature vector of each word. For the category k in the current task, the present invention first obtains the representation of all text segments marked as category k. For each text segment s ij , the feature vector h of its head word i and the feature vector h of the tail word j Splice to get the unit representation of the text segment

[0073]

[0074] Next, for entity category k, the K-means algorithm is used to cluster all text representations belonging to category k to obtain L clusters (L is set to 3, 4, 5, 6, 10 or 16, etc.). For each cluster i, the cluster centroid determined by the K-means algorithm represents the prototype representation of the cluster, so for entity category k, the present invention obtains L prototype representations. These prototype representations are initialized by the following formula:

[0075] p k,i =centroid(cluster i )fori=1,…,L

[0076] Among them, cluster i represents the i-th cluster, centroid(cluster i ) Calculate the centroid of the cluster as the i-th prototype p of entity category k k,i .

[0077] In addition, for all text representations of the k-th entity category, the present invention performs equal-proportion sampling on each cluster obtained by the K-means algorithm, retains the original samples corresponding to the sampled text representations, and constructs the core set of the category. This core set will be used for the subsequent knowledge distillation process and further training of the prototype network gating mechanism to enhance the robustness and adaptability of the model when processing new and old knowledge. These prototype representations represent the typical features or patterns of each category, and the core set saves samples of different features to better avoid catastrophic forgetting. Both the prototype representation and the core set can be updated or expanded according to the newly learned data, thereby naturally supporting the addition of new categories without the need to reconstruct the entire model. In the face of the addition of new categories, the framework of the present invention only needs to add expert networks and new prototype representations of new categories, which can flexibly adapt to the dynamic increase of categories and has strong scalability. In this way, the memory of old knowledge can be maintained, and the addition of new knowledge can be efficiently adapted.

[0078] Step A4: Execute the training of the review phase, including: obtaining a mixed training set consisting of a proprietary training set of the newly added category and a core set of each category, and updating the parameters of each expert network and encoding module using a preset total loss function, prototype representation and mixed training set training.

[0079] According to one embodiment of the present invention, in the review stage, in order to consolidate the knowledge learned by the expert network of the newly added category, the exclusive training set of the newly added category is temporarily retained, and combined with the core set of each category to form a mixed training set; the preset total loss function, the prototype representation of multiple categories and the mixed training set are used to train and update the parameters of each expert network and encoding module.

[0080] According to one embodiment of the present invention, in the review phase, the total loss function is set as the weighted sum of multiple sub-loss functions to update the parameters of the model from multiple aspects of knowledge and improve the performance of the model. Optionally, the total loss function is expressed as:

[0081]

[0082]

[0083] in, represents the total loss function, Represents the binary cross entropy loss function corresponding to the newly created expert network, L review Represents the distillation loss function corresponding to the expert network of non-new categories, represents the contrast loss function, λ1 represents The weight of The weight of The implementer can set the size of each weight as needed to balance the influence of each sub-loss function.

[0084] According to one embodiment of the present invention, in the review stage, for the newly created expert network, the input text comes from a proprietary training set, and the preset total loss function includes a binary cross entropy loss function for guiding the learning of the newly created expert network:

[0085]

[0086] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p′(k|s ij ) represents the kth expert network set in the review phase to identify unit s ij The predicted value of the output, where:

[0087]

[0088]

[0089] r ij =W6·ReLU(W5·e ij +b1)+b2

[0090] r k =W6·ReLU(W5·p k +b1)+b2

[0091] Where sigmoid(·) represents the sigmoid function, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection, r k Indicates the use of projection network to represent the prototype p k The vector obtained by projection, W5 represents the weight parameter of the first layer of the projection network, b1 represents the bias parameter of the first layer of the projection network, W6 represents the weight parameter of the second layer of the projection network, b2 represents the bias parameter of the second layer of the projection network, ReLU represents the ReLU activation function, Indicates that ij The prototypes of the same category and closest distance represent: Indicates e ij The shortest distance to the prototype representation of the same category, Indicates e ij is the shortest distance between the prototype representation of category l, exp(·) represents the exponential function, and K represents the total number of categories.

[0092] As can be seen, in the preview phase, the expert network is first In the form of prediction, we first focus on learning the knowledge of our own category, and then modify the prediction value of the expert network to By focusing on multiple aspects of knowledge (such as knowledge of its own category, knowledge involved in core set and prototype representation, and knowledge learned by comparison with other categories), it can improve its overall reasoning ability and accuracy when facing new category data and alleviate the forgetting of old knowledge; the knowledge of prototype representation and core set can be used to better improve the performance of expert network prediction.

[0093] For the expert network of non-new categories, the input text comes from the core set corresponding to the expert network, and the distillation loss function in the review stage is:

[0094]

[0095] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, n represents the total number of words in the input text, and K represents the total number of categories. Indicates the soft distillation tag that needs to be imitated. is the predicted value output by the model after the previous incremental learning of the class (equivalent to using the model before the new category is added as the teacher model and outputting a as the soft distillation label that needs to be imitated), p(k|s ij ) represents the kth expert network corresponding to the non-new category for the recognition unit s in the review stage ij The predicted value of the output,

[0096] According to one embodiment of the present invention, the total loss function includes a weighted sum of a binary cross entropy loss function, a distillation loss function, and a contrast loss function, and the parameters of each expert network, encoding network, and projection network are optimized according to the total loss function to reduce the total loss, wherein the contrast loss function is:

[0097]

[0098] Where N represents the total number of recognition units, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection is Indicates that ij The prototypes of the same category and closest distance represent: Indicates category l and e ij The nearest prototype represents,τ1 represents the preset temperature hyperparameter.

[0099] Optionally, in some cases, the projection network can be pre-trained and the parameters of the projection network can be frozen during the review phase, so that the contrastive loss function does not need to be involved in the total loss function, and the parameters of the projection network do not need to be updated when the parameters are updated.

[0100] This embodiment also provides a class incremental knowledge acquisition method based on the prototype network gating mechanism, including: obtaining the text to be predicted; inputting the text to be predicted into the encoding module trained according to the above method to obtain the feature vector of each word unit, and using the feature vector of each word unit to construct the unit representation of each paragraph; using the preset prototype network gating mechanism to determine the category to which each paragraph belongs, including: obtaining the similarity between the unit representation of each paragraph and the prototype representation of each category, selecting partially matching expert networks according to the similarity to predict the paragraph, and obtaining the prediction value of the selected expert network for whether the paragraph belongs to the category corresponding to the expert network; for overlapping paragraphs, only the prediction value of the paragraph with the highest prediction value is retained, and when the highest prediction value is greater than or equal to a preset threshold, it is judged that the retained paragraph belongs to the category corresponding to the expert network that outputs the highest prediction value; if the prediction value of a paragraph in all categories is less than the preset threshold τ2, then the paragraph does not belong to the category corresponding to any expert network. When predicting, the prediction value output by any k-th expert network is In addition, when predicting, the implementer can set the number of expert networks to be matched each time (such as 2, 3, 5, 8, 10, etc.) according to the performance of the computing device and the current number of categories. Assuming that there are currently 6 expert networks, the number of expert networks to be matched each time is set to 3; then for each paragraph, the similarity between the unit representation of the paragraph and the prototype representation of each category is calculated each time, and then 3 matching expert networks are selected for each paragraph according to the similarity from high to low, and the selected 3 matching expert networks are used to predict the paragraph.

[0101] Implementation Method 2

[0102] In the relationship extraction task, the goal of the task is to identify the relationship categories between entities. There are many types of relationship categories, which are formulated according to the needs of the implementer, such as: person relationship, organizational relationship, causal relationship, participation relationship or attribute relationship.

[0103] For example, in the existing relationship extraction dataset SemEval-2010 Task 8, the following character relationship categories are defined: Friendship, Family, Marriage, Enemies, Colleagues, and Romantic.

[0104] To avoid redundancy, the following focuses on the differences between the implementation details of the relationship extraction task and implementation mode 1. For the contents that do not point out the differences, please refer to implementation mode 1.

[0105] Difference 1: Definition of knowledge unit: a knowledge unit is any two entities in the input text whose relationship needs to be identified. ij , that is, the word-unit numbered i and the word-unit numbered j constitute a knowledge unit, where the word-unit numbered i and the word-unit numbered j are both entities. Each knowledge unit contains only two components (i≠j).

[0106] Difference 2: Unit representation of knowledge units Among them, H′0 represents the feature vector of the [CLS] tag in the BERT model, and h i The feature vector representing the head entity in the knowledge unit, h j A feature vector representing the tail entity in a knowledge unit.

[0107] Difference 3: As the unit representation changes, the corresponding prototype representation will also change. As the unit representation increases H′0, the dimension of the prototype representation will increase accordingly.

[0108] Difference 4: The structure of the expert network is different. The expert network is set to H rel =W3·concat(H′0,h i ,h j )+b3, where W3 represents the weight parameter, concat represents concatenation, and b3 represents the bias parameter. Compared with the entity recognition task that uses two FFN1 as the expert network, for the kth relationship category, only one FFN (i.e., W3 and b3) is needed as the expert, and the kth expert network is responsible for predicting whether the knowledge unit belongs to the kth relationship category.

[0109] In general, the technical solution of the method of the present invention has the advantage of practicality. First, in view of the forward incompatibility problem in incremental class learning, the model can avoid losing its memory of old knowledge when learning new knowledge through the setting, dynamic adjustment and optimization of prototype representation and core set, thereby improving the prediction performance of old categories. Secondly, through the multi-expert hybrid prediction method, the entity features of multiple categories are comprehensively processed to improve the accuracy of new category data and solve the problem of backward incompatibility. In addition, the prototype network gating mechanism used by the method of the present invention can flexibly adapt to the dynamic increase of categories and has strong scalability.

[0110] It should be noted that although the above describes the various steps in a specific order, it does not mean that the various steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0111] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0112] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0113] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A class incremental learning method in the field of knowledge acquisition, which is executed each time a new class needs to be recognized in a knowledge acquisition task, comprising: S1. Create a dedicated expert network for the newly added category in the model, the model includes a BERT-based encoding module for extracting feature vectors of each word from the input text and an expert network for each category; the expert network for each category is used to extract a category embedding from a unit representation of a knowledge unit in the input text, and determine a prediction value that the knowledge unit belongs to the category based on the category embedding, wherein the unit representation includes the feature vector of the word involved in the knowledge unit; S2, performing training in the preview stage, including: obtaining the proprietary training set used in S1 for training whether the knowledge unit belongs to the newly added category, using the proprietary training set and the binary cross entropy loss function to train the newly created expert network and encoding module, and obtaining the previewed encoding network and expert network; S3. Using the latest encoding network to extract feature vectors from each word in the text of the proprietary training set of each category, to construct the unit representation of the knowledge unit, and clustering the unit representation of the knowledge unit of each category to obtain multiple clusters, determining the representative prototype representation and extracted unit representation from each cluster, and using the original sample corresponding to the extracted unit representation to construct the core set of the category; S4. Execute the training in the review phase, including: obtaining a mixed training set consisting of a proprietary training set for the newly added category and a core set for each category, and updating the parameters of each expert network and encoding module using a preset total loss function, prototype representation, and mixed training set training.

2. The method according to claim 1, characterized in that For each category, the representative prototype representation and the extraction unit representation from each cluster are determined by: Obtaining a unit representation of each recognition unit belonging to the category; The unit representations of the category are clustered respectively using a preset clustering algorithm to obtain multiple clusters under the category; Using the centroid of each cluster under the category as the prototype representation of the cluster, multiple prototype representations under the category are obtained; An equal proportion of sampling is performed from each cluster under this category to obtain the extracted unit representation.

3. The method according to claim 2, characterized in that The binary cross entropy loss function in the preview stage is: Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p(k|s ij ) represents the k-th expert network for the recognition unit s ij Output prediction value, prediction value in the preview phase Represents knowledge units ij , sigmoid(·) represents the sigmoid function.

4. The method according to claim 3, characterized in that: In the review phase, for the newly created expert network, the input text comes from a proprietary training set, and the preset total loss function includes a binary cross entropy loss function used to guide the learning of the newly created expert network: Among them, p′(k|s ij ) represents the kth expert network set in the review phase to identify unit s ij The predicted value of the output, where: <h2 style=";text-align:left;direction:ltr">r<h2 style=";text-align:left;direction:ltr"> ij <h2 style=";text-align:left;direction:ltr"> =W6·ReLU(W5·e<h2 style=";text-align:left;direction:ltr"> ij <h2 style=";text-align:left;direction:ltr"> +b1)+b2 <h2 style=";text-align:left;direction:ltr">r<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> =W6·ReLU(W5·p<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> +b1)+b2 Among them, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection, r k Indicates the use of projection network to represent the prototype p k The vector obtained by projection, W5 represents the weight parameter of the first layer of the projection network, b1 represents the bias parameter of the first layer of the projection network, W6 represents the weight parameter of the second layer of the projection network, b2 represents the bias parameter of the second layer of the projection network, ReLU represents the ReLU activation function, Indicates that ij The prototypes of the same category and closest distance represent: Indicates e ij The shortest distance to the prototype representation of the same category, Indicates e ij is the shortest distance between the prototype representation of category l, exp(·) represents the exponential function, and K represents the total number of categories.

5. The method according to claim 4, characterized in that For the expert network of non-new categories, the input text comes from the core set corresponding to the expert network, and the distillation loss function in the review stage is: in, Indicates the soft distillation tag that needs to be imitated. is the predicted value output by the model after the previous class incremental learning, p(k|s ij ) represents the kth expert network corresponding to the non-new category for the recognition unit s in the review stage ij The predicted value of the output, 6. The method according to claim 5, characterized in that The total loss function includes the weighted sum of the binary cross entropy loss function, the distillation loss function, and the contrast loss function. According to the total loss function, the parameters of each expert network, the encoding network, and the projection network are optimized to reduce the total loss. The contrast loss function is: Where N represents the total number of recognition units and τ1 represents the preset temperature hyperparameter.

7. The method according to claim 6, characterized in that The knowledge acquisition task is a named entity recognition task or a relationship extraction task, wherein the recognition unit under the named entity recognition task is the word-gram at the head and the word-gram at the tail of a paragraph in the input text, and the recognition unit under the relationship extraction task is the word-gram corresponding to two entities of the relationship category to be identified in the input text.

8. A quasi-incremental knowledge acquisition method based on a prototype network gating mechanism, characterized in that: include: Get the text to be predicted; Inputting the text to be predicted into the encoding module trained according to the method of any one of claims 1 to 7, obtaining the feature vector of each word unit, and constructing the unit representation of each knowledge unit using the feature vector of each word unit; The preset prototype network gating mechanism is used to determine the category to which each knowledge unit belongs, including: Obtain the similarity between the unit representation of each knowledge unit and the prototype representation of each category, select partially matched expert networks based on the similarity to predict the knowledge unit, and obtain the prediction value of whether the knowledge unit belongs to the corresponding category of the expert network. For overlapping knowledge units, only the prediction value of the knowledge unit with the highest prediction value is retained, and when the highest prediction value is greater than or equal to a preset threshold, it is determined that the retained knowledge unit belongs to the category corresponding to the expert network that outputs the highest prediction value; If the predicted values ​​of a knowledge unit in all categories are less than the preset threshold, the knowledge unit does not belong to the category corresponding to any expert network.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.

Images