A prototype network gate mechanism-based incremental knowledge acquisition method

Abstract

The application provides a kind of class incremental knowledge acquisition method based on prototype network gate mechanism, including obtaining text to be predicted;Text is input into the encoding module trained by the class incremental learning method in the knowledge acquisition field, the feature vector of each word element is obtained, and the feature vector of each word element is used to construct the unit representation of each knowledge unit;Determine the category to which each knowledge unit belongs using the preset prototype network gate mechanism, including: obtaining the similarity between the unit representation of each knowledge unit and the prototype representation of each category, predicting the knowledge unit according to the similarity of the screened expert network, obtaining the prediction value of the screened expert network to predict whether the knowledge unit belongs to the category corresponding to the expert network;For overlapping knowledge units, only the prediction value of the knowledge unit with the highest prediction value is retained;If the prediction value of a knowledge unit in all categories is less than a preset threshold, the knowledge unit does not belong to any category corresponding to the expert network.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, specifically to the field of incremental learning, and more specifically to a quasi-incremental knowledge acquisition method based on a prototype network gating mechanism. Background Art

[0002] As the internet plays an increasingly important role in our daily lives, the vast amount of unstructured text data poses a significant challenge to people's access to information. Helping people quickly acquire useful knowledge has become a pressing issue. Against this backdrop, information extraction technology has emerged as a key task in natural language processing. Information extraction technology is primarily used to extract useful knowledge, such as events and entities, from unstructured text to facilitate knowledge graph construction, information retrieval, and other natural language processing tasks. Event extraction, a key research area within information extraction, focuses on extracting events of interest from unstructured text and presenting them in a structured manner.

[0003] Named Entity Recognition (NER) and Relation Extraction (RE) tasks occupy a core position in the field of knowledge acquisition and are crucial for understanding and extracting key information from text. However, when traditional knowledge acquisition models perform incremental learning on new categories of data, people hope that the model can gradually adapt to the new categories of data without losing predictive performance on the old categories. However, in reality, the model tends to lose its memory of old knowledge while learning new knowledge; this phenomenon is particularly evident in quasi-incremental learning environments. To address this challenge, the model must not only be able to flexibly adapt to new data, but also maintain a stable memory of the learned knowledge during the continuous learning process.

[0004] Incremental learning for NER tasks is more complex than for RE tasks. The challenges faced by incremental learning for NER tasks mainly stem from the particularity of the task, especially when dealing with backward incompatibility and forward incompatibility. The following uses NER tasks as an example to discuss these two incompatibilities and their impact on NER tasks:

[0005] 1) Forward incompatibility:

[0006] In the quasi-incremental learning scenario of the NER task, when new entity types are introduced for learning, the references of previously learned entity types may still appear in the training samples of the current task, but these samples do not provide corresponding annotations for these references. In this case, the model may forget the previously learned entity types or fail to correctly identify these unlabeled entity references because they do not appear as points of interest in the training data of the current task. This leads to the so-called catastrophic forgetting problem, that is, the model loses or confuses old knowledge when learning new knowledge.

[0007] 2) Backward incompatibility:

[0008] In the current learning task, some non-entity references may be identified as a certain entity type in future tasks. This means that during the training phase of the current task, these references are considered non-entities, but in future learning tasks, they need to be identified and correctly classified. In this case, the model needs to be able to adapt to the new entity type and re-evaluate and identify these references that were previously considered non-entities in future learning tasks. The present invention calls this phenomenon "belief fixation". In psychology, belief fixation means that once a certain idea is formed, it is difficult to change the original belief even in the face of evidence that directly refutes the idea. Even if the wrong idea is corrected, people will still be affected by the original wrong information. This is because the original wrong information has formed an impression in their cognitive structure, and changing this impression requires more cognitive effort than accepting new information.

[0009] Existing research on incremental learning in the field of knowledge acquisition mainly focuses on backward incompatibility. Knowledge distillation is usually adopted to use the previously learned model as the teacher model, predict the distilled pseudo-labels on the current sample, and then jointly learn the current student model through these labels and the current true labels. However, the problem of belief fixation is ignored, resulting in the need to improve the accuracy of knowledge acquisition.

[0010] It can be seen that in the field of knowledge acquisition, the existing technology has not effectively solved the problems of forward incompatibility and backward incompatibility.

[0011] It should be noted that this background information is intended solely to introduce relevant information related to the present invention to facilitate understanding of the present invention's technical solution. It does not necessarily constitute prior art. Relevant information submitted and disclosed together with the present invention's solution should not be considered prior art unless there is evidence that the relevant information was disclosed prior to the filing date of the present invention. Summary of the Invention

[0012] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a quasi-incremental knowledge acquisition method based on the prototype network gating mechanism.

[0013] The purpose of the present invention is achieved through the following technical solutions:

[0014] According to a first aspect of the present invention, a class incremental learning method is provided, which is executed each time a new category needs to be recognized in a knowledge acquisition task, comprising: S1, creating a dedicated expert network for the newly added category in the model, the model comprising a BERT-based encoding module for extracting feature vectors of each word unit from an input text and an expert network for each category; the expert network for each category is used to extract category embeddings from unit representations of knowledge units in the input text, and to determine a predicted value of whether the knowledge unit belongs to the category based on the category embeddings, wherein the unit representation includes feature vectors of word units involved in the knowledge unit; S2, performing training in the preview stage, comprising: obtaining a dedicated training set in S1 for training whether the knowledge unit belongs to the newly added category, and using the dedicated training set and the binary intersection The cross entropy loss function is used to train the newly created expert network and encoding module to obtain the previewed encoding network and expert network; S3, using the latest encoding network to extract feature vectors for each word in the text of the proprietary training set of each category to construct the unit representation of the knowledge unit, and clustering the unit representation of the knowledge unit of each category to obtain multiple clusters, and determining the representative prototype representation and extracted unit representation from each cluster, and using the original samples corresponding to the extracted unit representation to construct the core set of the category; S4, executing the review stage training, including: obtaining a mixed training set consisting of the proprietary training set of the newly added category and the core set of each category, and using the preset total loss function, prototype representation and mixed training set training to update the parameters of each expert network and encoding module.

[0015] Optionally, for each category, the method of determining a representative prototype representation and extracted unit representation from each cluster includes: obtaining the unit representation corresponding to each recognition unit belonging to the category; clustering the unit representations of the category using a preset clustering algorithm to obtain multiple clusters under the category; using the centroid of each cluster under the category as the prototype representation of the cluster to obtain multiple prototype representations under the category; and sampling in equal proportion from each cluster under the category to obtain extracted unit representations.

[0016] Optionally, the binary cross entropy loss function in the preview phase is:

[0017]

[0018] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p(k|s ij) represents the k-th expert network for the recognition unit s ij Output prediction value, prediction value in the preview phase Represents knowledge units ij , sigmoid(·) represents the sigmoid function.

[0019] Optionally, during the review phase, for the newly created expert network, whose input text comes from a proprietary training set, the preset total loss function includes a binary cross entropy loss function used to guide the learning of the newly created expert network:

[0020]

[0021] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p′(k|s ij ) represents the kth expert network set in the review phase to identify the unit s ij The predicted value of the output, where:

[0022]

[0023]

[0024] r ij =W6·ReLU)W5·e ij +b1)+b2

[0025] r k =W6·ReLU(W5·p k +b1)+b2

[0026] Among them, sigmoid(·) represents the sigmoid function, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection, r k Indicates that the prototype representation p is represented by the projection network k The vector obtained by projection, W5 represents the weight parameter of the first layer of the projection network, b1 represents the bias parameter of the first layer of the projection network, W6 represents the weight parameter of the second layer of the projection network, b2 represents the bias parameter of the second layer of the projection network, ReLU represents the ReLU activation function, Indicates e ik The prototypes of the same category and closest distance are represented by: Indicates eik The shortest distance to the prototype representation of the same category, Indicates e ik The shortest distance between the prototype representation of category l, exp(·) represents the exponential function, and K represents the total number of categories.

[0027] Optionally, for the expert network of non-new categories, the input text comes from the core set corresponding to the expert network, and the distillation loss function in the review stage is:

[0028]

[0029] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, n represents the total number of words in the input text, and K represents the total number of categories. Indicates the soft distillation tag that needs to be imitated. is the predicted value output by the model after the previous class incremental learning, p(k|s ij ) represents the k-th expert network corresponding to the non-new category for the recognition unit s in the review phase ij The predicted value of the output, Optionally, the total loss function includes a weighted sum of a binary cross entropy loss function, a distillation loss function, and a contrast loss function. The parameters of each expert network, encoding network, and projection network are optimized according to the total loss function to reduce the total loss. The contrast loss function is:

[0030]

[0031] Where N represents the total number of recognition units, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection is Indicates e ij The prototypes of the same category and closest distance are represented by: Indicates category l and e ik The nearest prototype represents the distance, and τ1 represents the preset temperature hyperparameter.

[0032] Optionally, the knowledge acquisition task is a named entity recognition task or a relationship extraction task, wherein the recognition units under the named entity recognition task are the head word and the tail word of a paragraph in the input text, and the recognition units under the relationship extraction task are the word units corresponding to two entities of the relationship category to be identified in the input text.

[0033] According to a second aspect of the present invention, a method for acquiring quasi-incremental knowledge based on a prototype network gating mechanism is provided, comprising: acquiring a text to be predicted; inputting the text to be predicted into an encoding module trained according to the method of the first aspect to obtain a feature vector of each word unit, and constructing a unit representation of each knowledge unit using the feature vector of each word unit; determining the category to which each knowledge unit belongs using a preset prototype network gating mechanism, comprising: acquiring the similarity between the unit representation of each knowledge unit and the prototype representation of each category, screening partially matching expert networks based on the similarity to predict the knowledge unit, and obtaining a prediction value of the screened expert network on whether the knowledge unit belongs to the category corresponding to the expert network; for overlapping knowledge units, retaining only the prediction value of the knowledge unit with the highest prediction value, and when the highest prediction value is greater than or equal to a preset threshold, judging that the retained knowledge unit belongs to the category corresponding to the expert network that outputs the highest prediction value; if the prediction values ​​of a knowledge unit in all categories are less than the preset threshold, then the knowledge unit does not belong to the category corresponding to any expert network.

[0034] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of the method described in the first aspect and / or the second aspect by executing the executable instructions. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0036] Figure 1 A schematic diagram of a flow chart of a quasi-incremental learning method according to an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of samples in a proprietary training set and samples in a regular training set according to an embodiment of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0039] As mentioned in the background technology section, in the field of knowledge acquisition, the existing technology has not solved the problems of forward incompatibility and backward incompatibility well. For the problem of forward incompatibility, the method of the present invention is based on the encoding module of BERT to capture the dependencies between each word in the input text; then a number of prototype representations and core sets are constructed for each category. These prototype representations and core sets can be continuously updated and optimized in each incremental learning to avoid the model losing its memory of old knowledge when learning new knowledge, thereby improving the prediction performance of old categories; secondly, for the problem of backward incompatibility, the method of the present invention implements a multi-expert hybrid prediction method by constructing an exclusive expert network for each newly added category. Each expert network can focus on learning the relevant knowledge of its corresponding category, improve the accuracy of predicting new category data, and solve the problem of backward incompatibility; thirdly, in the preview stage, the expert network is first allowed to focus on learning the knowledge of its own category in the proprietary training set, and then in the review stage, the prediction performance of each expert network is improved with the help of the knowledge of the newly added type's proprietary training set, prototype representation and core set.

[0040] The knowledge acquisition tasks involved in the method of the present invention include named entity recognition and relationship extraction. Implementation 1 below uses the implementation scenario of the named entity recognition task as an example. Implementation 2 introduces the implementation scenario of the relationship extraction task, but to avoid redundancy, it focuses on the differences from Implementation 1.

[0041] Implementation Method 1

[0042] In the named entity recognition task, the goal of the task is to identify entity categories. It is necessary to identify whether each component element in the input text (such as characters, words, and phrases, and the division unit of the component elements depends on the type of natural language) is a non-entity category or a certain entity category.

[0043] For example: suppose the categories that the task needs to identify include:

[0044] O: represents an ordinary word or phrase, not belonging to any named entity.

[0045] PER: stands for Person, which is used to mark the names of people or noun phrases related to people in the text.

[0046] ORG: stands for Organization, and is used to mark the organization name in the text, such as company, institution, group, etc.

[0047] LOC: stands for Location, and is used to mark geographical location names in text, such as cities, countries, rivers, mountains, etc.

[0048] MISC: stands for Miscellaneous category, which is used to mark other types of named entities that do not belong to the above PER, ORG, and LOC categories, such as work names, event names, proper nouns, etc.

[0049] Among them, O is a non-entity category, and PER, ORG, LOC, and MISC are entity categories.

[0050] Of course, other named entity recognition tasks may be possible based on the needs of the implementer. For example, another task is to label each word in each sentence of the input text as B, I, E, S, or O, where "O" indicates that the corresponding word is not an entity (non-entity category), "S" indicates that the corresponding word is a separate entity, and "B", "I", and "E" indicate that the corresponding word is the beginning, middle, and end of an entity, respectively.

[0051] According to one embodiment of the present invention, see Figure 1 A class incremental learning method is executed each time a new class needs to be recognized in a knowledge acquisition task. The method includes steps A1, A2, A3, and A4. To better understand the present invention, each step is described in detail below in conjunction with specific embodiments.

[0052] Step A1: Create a dedicated expert network for the newly added category in the model. The model includes a BERT-based encoding module for extracting feature vectors of each word from the input text and an expert network for each category; the expert network for each category is used to extract category embeddings from the unit representation of the knowledge unit in the input text, and determine the predicted value of the knowledge unit belonging to the category based on the category embedding, where the unit representation includes the feature vectors of the word involved in the knowledge unit.

[0053] For existing named entity recognition tasks, after using the BERT model, it is necessary to add a conditional random field (CRF) and the Viterbi algorithm to complete named entity recognition. This is equivalent to having a single expert complete the labeling of all categories. Therefore, it is easy to cause forward and backward incompatibility issues. Therefore, the embodiments of the present invention provide improvements to this problem.

[0054] According to one embodiment of the present invention, the model is a named entity recognition model. When initially constructing the named entity recognition model, X categories of named entity recognition known to the implementer at the time can be used as the named entity recognition model, and X expert networks can be added after the encoding module. The encoding module can use the BERT model; the expert network can be constructed using a neural network, such as a feedforward neural network. The BERT model and the X expert networks are then trained based on a training set with relevant annotated labels. Each expert network is dedicated to the recognition of a category. If the implementer needs to add a new category, a new dedicated expert network is created for each new category in addition to the X expert networks.

[0055] The encoding module uses the BERT model as a context encoder to capture the dependencies between the various components in the input text. n ] to represent the input text sequence X, x1~x n The n elements corresponding to encoding 1 to n (corresponding to n word tokens) are then fed into the BERT encoder to obtain the contextualized hidden vector H corresponding to the input text. The process can be expressed as:

[0056] H=BERT(X)

[0057] in, Among them, h1~h n The feature vectors of the n components corresponding to encoding 1 to n, d j Represents the dimension of the feature vector.

[0058] According to one embodiment of the present invention, the knowledge unit is set as a paragraph in a sentence, which can be represented as s ij , i.e. a paragraph starting with word number i and ending with word number j. Each paragraph may contain one component element (i=j), or may contain multiple components (two or more, i<j). In the embodiment of the present invention, if a knowledge unit is finally identified as a certain category, it means that the knowledge unit (paragraph s ij ) is identified as the category. The unit representation of the knowledge unit is set as the feature vector of the token number i and the feature vector of the token number j: (h i , h j ), where h i represents the feature vector of the word number i extracted by the encoding module, h j Represents the feature vector of the word number j extracted by the encoding module.

[0059] According to one embodiment of the present invention, the expert network is configured to use two feedforward neural networks (FFNs) with different feedforwards to process the feature vectors of two word units in the unit representation of the knowledge unit to extract category embeddings, which are expressed as:

[0060]

[0061] in, Represents the expert network of category k for the text segment s ij The extracted category embedding, s, e, k represent the start position, end position, and kth entity category respectively, FFN s,k Indicates the text segment ij The feedforward neural network, FFN, processes the feature vector of the word number i in e,k Indicates the text segment ij The feedforward neural network that processes the feature vector of the word number j in d o FFN s,k and FFN e,k The output dimension of k is . Modeling the clear separation of each entity category in each paragraph can promote learning and knowledge distillation and reduce the interference between multiple tasks. Organize into a matrix The upper triangular area of ​​​​is stored, where The row and column indicate the start and end respectively.

[0062] Therefore, the feedforward neural network FFN of the k-th entity category can be s,k and FFN e,k As "experts" of this category, the "experts" provide the model with profound insights into the recognition of a certain category. Through the integration of these experts, the method of the present invention can provide customized processing strategies for different entity categories.

[0063] Step A2: Execute the training in the preview stage, including: obtaining the proprietary training set in S1 for training whether the knowledge unit belongs to the newly added category, using the proprietary training set and the binary cross entropy loss function to train the newly created expert network and encoding module to obtain the previewed encoding network and expert network.

[0064] According to one embodiment of the present invention, a dedicated training set may include multiple samples, each of which is a sentence or a paragraph of text; the label information in the dedicated training set indicates whether each component element in each sample belongs to the category to be identified by the corresponding expert network. For example, for a certain component element, if the corresponding value in the label information is 1, it means it belongs to the category, and if it is 0, it means it does not belong to the category. The input text can be a single sample; or it can be a combination of multiple samples as the input text. It should be noted that in conventional named entity recognition datasets, different categories are marked on the samples at the same time. See Figure 2 , illustrating the difference between specialized training sets and conventional datasets. In conventional datasets, the label information for a sample contains all possible class labels. For example, for the sample "Emma at NASA launched asatellite," the true value of the label T is "PER O ORG OOO," indicating that the first element in the sample is a person's name, the third element is an organization's name, and the remaining elements are non-entity types. However, when the specialized training set is changed to a specific category, only elements belonging to that category are labeled as that category (indicated by 1). All other categories can be labeled as non-entity categories (or other categories, indicated by 0), as shown in labels T1 and T2.

[0065] In addition, when it is desired to simulate the method of the present invention with an existing data set, a conventional data set, such as the CoNLL-2003 data set or the Ontonotes-5.0 data set, can be used for simulation. For example: first, based on the conventional data set, prepare a dedicated training set for each part; then select some categories from all the entity categories contained in the data set and train a model first; then, each time, select a category from the other unselected categories as a new category, implement the method of the present invention to observe the effect. However, in actual scenarios, the implementer can train a model based on the dedicated training set of all currently known categories. Then, if a new category is to be added, samples are labeled for the new category to construct a dedicated training set; and a new expert network is created to be trained according to the method of the present invention.

[0066] According to one embodiment of the present invention, in the preview phase, only classification loss is used to supervise the training of the model. Since the expert network of each network is exclusive, each expert network is a binary classification network, and the corresponding binary classification cross entropy loss function is used for simplicity. Preferably, the binary classification cross entropy loss function in the preview phase is:

[0067]

[0068] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p(k|s ij ) represents the k-th expert network for the recognition unit s ij Output prediction value, prediction value in the preview phase Represents knowledge units ij , where sigmoid(·) represents the sigmoid function. During the pre-training phase, the parameter update method can be customized by the implementer. For example, the implementer can set a first learning rate for the newly created expert network and a second learning rate for the encoding module, where the second learning rate is much smaller than the first learning rate, to fine-tune the encoding module during pre-training. Alternatively, if the implementer wishes to only change the parameters of the newly created expert network, the parameters of the encoding module can be frozen.

[0069] Step A3: Utilize the latest encoding network to extract feature vectors from each word in the text of the dedicated training set for each category. This is used to construct unit representations of knowledge units. The unit representations of each category's knowledge units are clustered to obtain multiple clusters. Representative prototype representations and extracted unit representations are determined within each cluster. The original samples corresponding to the extracted unit representations are used to construct the core set for that category.

[0070] Although the previous steps set up a dedicated expert network for each category, effectively routing and integrating these expert predictions becomes a challenge when the number of categories increases dynamically. Using a neural network routing module with a fixed output length would be difficult to scale, and retraining the routing module at each incremental learning step would inevitably introduce new catastrophic forgetting issues in the routing module.

[0071] According to one embodiment of the present invention, for each category, the method of determining a representative prototype representation and an extracted unit representation from each cluster includes: obtaining the unit representation corresponding to each recognition unit belonging to the category; clustering the unit representations of the category using a preset clustering algorithm to obtain multiple clusters under the category; using the centroid of each cluster in the multiple clusters under the category as the prototype representation of the cluster to obtain multiple prototype representations under the category; sampling in equal proportions from each cluster under the category to obtain the extracted unit representation. The identification units belonging to a certain category can be determined by extracting the text segments belonging to the category from the proprietary training set with the help of annotated labels, and then determining the unit representation of this part of the text segment. Subsequently, based on the text segment, a preset clustering algorithm (such as the K-means algorithm) is used to cluster the text segments belonging to the category.

[0072] Indicatively, after completing pre-training, the trained model is M k , using the latest model to extract the feature vector of each word. For the category k in the current task, the present invention first obtains the representation of all text segments marked as category k. For each text segment s ij , the feature vector h of its head word i and the feature vector h of the tail word j Splice and get the unit representation of the paragraph

[0073]

[0074] Next, for entity category k, the K-means algorithm is used to cluster all text representations belonging to category k, obtaining L clusters (L is set to 3, 4, 5, 6, 10, or 16, for example). For each cluster i, the cluster centroid determined by the K-means algorithm represents the prototype representation of the cluster. Therefore, for entity category k, the present invention obtains L prototype representations. These prototype representations are initialized by the following formula:

[0075] p k,i =centroid(cluster i )fori=1,…,L

[0076] Among them, cluster i Indicates the i-th cluster, centroid (cluster i ) Calculate the centroid of the cluster as the i-th prototype p of entity category k k,i .

[0077] In addition, for all text representations of the k-th entity category, the present invention performs equal-proportion sampling on each cluster obtained by the K-means algorithm, retains the original samples corresponding to the sampled text representations, and constructs the core set of the category. This core set will be used for the subsequent knowledge distillation process and further training of the prototype network gating mechanism to enhance the robustness and adaptability of the model when processing new and old knowledge. These prototype representations represent the typical features or patterns of each category, and the core set saves samples of different features to better avoid catastrophic forgetting. Both the prototype representation and the core set can be updated or expanded based on the newly learned data, thereby naturally supporting the addition of new categories without the need to reconstruct the entire model. In the face of the addition of new categories, the framework of the present invention only needs to add expert networks and new prototype representations of new categories, which can flexibly adapt to the dynamic increase of categories and has strong scalability. In this way, it can not only maintain the memory of old knowledge, but also efficiently adapt to the addition of new knowledge.

[0078] Step A4: Execute the review phase training, including: obtaining a mixed training set consisting of a dedicated training set for the newly added category and a core set for each category, and updating the parameters of each expert network and encoding module using the preset total loss function, prototype representation, and mixed training set training.

[0079] According to one embodiment of the present invention, during the review phase, in order to consolidate the knowledge learned by the expert network of the newly added category, the dedicated training set of the newly added category is temporarily retained, and combined with the core set of each category to form a mixed training set; the preset total loss function, the prototype representation of multiple categories and the mixed training set are used to train and update the parameters of each expert network and encoding module.

[0080] According to one embodiment of the present invention, during the review phase, the total loss function is set as the weighted sum of multiple sub-loss functions to update the model parameters based on knowledge from multiple aspects and improve the performance of the model. Optionally, the total loss function is expressed as:

[0081]

[0082]

[0083] in, represents the total loss function, Represents the binary cross entropy loss function corresponding to the newly created expert network, L review Represents the distillation loss function corresponding to the expert network of non-new categories, represents the contrast loss function, λ1 represents The weight of λ2 represents The weight of λ3 represents The implementer can set the size of each weight as needed to balance the influence of each sub-loss function.

[0084] According to one embodiment of the present invention, during the review phase, for the newly created expert network, whose input text comes from a proprietary training set, the preset total loss function includes a binary cross entropy loss function for guiding the learning of the newly created expert network:

[0085]

[0086] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p′(k|s ij ) represents the kth expert network set in the review phase to identify the unit s ij The predicted value of the output, where:

[0087]

[0088]

[0089] r ij =W6·ReLU(W5·e ij +b1)+b2

[0090] r k =W6·ReLU(W5·p k +b1)+b2

[0091] Among them, sigmoid(·) represents the sigmoid function, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection, r k Indicates that the prototype representation p is represented by the projection network k The vector obtained by projection, W5 represents the weight parameter of the first layer of the projection network, b1 represents the bias parameter of the first layer of the projection network, W6 represents the weight parameter of the second layer of the projection network, b2 represents the bias parameter of the second layer of the projection network, ReLU represents the ReLU activation function, Indicates e ij The prototypes of the same category and closest distance are represented by: Indicates e ij The shortest distance to the prototype representation of the same category, Indicates e ij The shortest distance between the prototype representation of category l, exp(·) represents the exponential function, and K represents the total number of categories.

[0092] As we can see, in the preview phase, the expert network is first First, we focus on learning the knowledge of our own category, and then modify the prediction value of the expert network to By focusing on multiple aspects of knowledge (such as knowledge of its own category, knowledge involved in core set and prototype representation, and knowledge learned by comparison with other categories), it can improve its overall reasoning ability and accuracy when facing new category data and alleviate the forgetting of old knowledge; the knowledge of prototype representation and core set can be used to better improve the performance of expert network prediction.

[0093] For the expert network of non-new categories, the input text comes from the core set corresponding to the expert network. The distillation loss function in the review stage is:

[0094]

[0095] Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, n represents the total number of words in the input text, and K represents the total number of categories. Indicates the soft distillation tag that needs to be imitated. The predicted value output by the model after the previous incremental learning of the class (equivalent to taking the model before the new category is added as the teacher model and outputting a as the soft distillation label that needs to be imitated), p(k|s ij ) represents the k-th expert network corresponding to the non-new category for the recognition unit s in the review phase ij The predicted value of the output,

[0096] According to one embodiment of the present invention, the total loss function includes a weighted sum of a binary cross entropy loss function, a distillation loss function, and a contrast loss function. The parameters of each expert network, encoding network, and projection network are optimized according to the total loss function to reduce the total loss. The contrast loss function is:

[0097]

[0098] Where N represents the total number of recognition units, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection is Indicates e ij The prototypes of the same category and closest distance are represented by: Indicates category l and e ij The nearest prototype represents the distance, and τ1 represents the preset temperature hyperparameter.

[0099] Alternatively, in some cases, the projection network can be pre-trained and its parameters can be frozen during the review phase, so that the contrastive loss function does not need to be involved in the total loss function, and the parameters of the projection network do not need to be updated during parameter updates.

[0100] This embodiment also provides a class incremental knowledge acquisition method based on the prototype network gating mechanism, including: obtaining the text to be predicted; inputting the text to be predicted into the encoding module trained according to the above method to obtain the feature vector of each word unit, and using the feature vector of each word unit to construct the unit representation of each paragraph; using the preset prototype network gating mechanism to determine the category to which each paragraph belongs, including: obtaining the similarity between the unit representation of each paragraph and the prototype representation of each category, screening the partially matched expert network according to the similarity to predict the paragraph, and obtaining the prediction value of the screened expert network on whether the paragraph belongs to the category corresponding to the expert network; for overlapping paragraphs, only the prediction value of the paragraph with the highest prediction value is retained, and when the highest prediction value is greater than or equal to the preset threshold, it is judged that the retained paragraph belongs to the category corresponding to the expert network that outputs the highest prediction value; if the prediction value of a paragraph in all categories is less than the preset threshold τ2, then the paragraph does not belong to the category corresponding to any expert network. During prediction, the prediction value output by any k-th expert network is In addition, when making predictions, implementers can set the number of expert networks to match each time (e.g., 2, 3, 5, 8, 10, etc.) based on the performance of the computing device and the current number of categories. Assuming there are currently 6 expert networks, and the number of expert networks matched each time is set to 3, then for each paragraph, the similarity between the unit representation of the paragraph and the prototype representation of each category is first calculated, and then 3 matching expert networks are selected for each paragraph in descending order of similarity. The 3 selected matching expert networks are used to make predictions for the paragraph.

[0101] Implementation Method 2

[0102] In the relationship extraction task, the goal is to identify the relationship categories between entities. There are many types of relationship categories, which are determined according to the needs of the implementer, such as: person relationship, organizational relationship, causal relationship, participation relationship, or attribute relationship.

[0103] For example, the existing relationship extraction dataset SemEval-2010 Task 8 defines the following character relationship categories: friendship, family, marriage, enemies, colleagues, and romantic.

[0104] To avoid redundancy, the following focuses on the differences between the implementation details of the relationship extraction task and implementation method 1. For content that does not point out the differences, please refer to implementation method 1.

[0105] Difference 1: Definition of knowledge unit: a knowledge unit is any two entities in the input text whose relationship needs to be identified. ij , that is, a knowledge unit is composed of a word-gram numbered i and a word-gram numbered j, where both word-grams numbered i and j are entities. Each knowledge unit contains only two components (i≠j).

[0106] Difference 2: Unit representation of knowledge units Among them, H′0 represents the feature vector of [CLS] tag in BERT model, h i The feature vector representing the head entity in the knowledge unit, h j A feature vector representing the tail entity in a knowledge unit.

[0107] Difference 3: Due to the change in the unit representation, the corresponding prototype representation will also change. Since the unit representation increases H′0, the dimensionality of the prototype representation will increase accordingly.

[0108] Difference 4: The structure of the expert network is different. The expert network is set to H rel =W3·concat(H′0,h i ,h j )+b3, where W3 represents the weight parameter, concat represents concatenation, and b3 represents the bias parameter. Compared to the entity recognition task that uses two FFN1s as expert networks, for the kth relation category, only one FFN (i.e., W3 and b3) is required as the expert. The kth expert network is responsible for predicting whether the knowledge unit belongs to the kth relation category.

[0109] In general, the technical solution of the method of the present invention has the advantage of practicality. First, it addresses the forward incompatibility problem in incremental class learning by setting, dynamically adjusting, and optimizing prototype representations and core sets to prevent the model from losing its memory of old knowledge when learning new knowledge, thereby improving the prediction performance of old categories. Secondly, through a multi-expert hybrid prediction method, entity features of multiple categories are comprehensively processed to improve the accuracy of new category data and solve the problem of backward incompatibility. In addition, the prototype network gating mechanism used by the method of the present invention can flexibly adapt to the dynamic increase of categories and has strong scalability.

[0110] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0111] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0112] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0113] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A class incremental learning method for the knowledge acquisition field, which is executed each time a new class needs to be recognized in a knowledge acquisition task, comprising: S1. Create a dedicated expert network for the newly added category in the model. The model includes a BERT-based encoding module for extracting feature vectors for each word from the input text and an expert network for each category. The expert network for each category is used to extract a category embedding from the unit representation of the knowledge unit in the input text, and to determine a prediction value for the knowledge unit belonging to the category based on the category embedding, where the unit representation includes the feature vector of the word involved in the knowledge unit. S2. Execute training in the preview phase, including: obtaining the proprietary training set used in S1 for training whether the knowledge unit belongs to the newly added category, and training the newly created expert network and encoding module using the proprietary training set and a binary cross entropy loss function to obtain a previewed encoding network and expert network; S3. Utilize the latest encoding network to extract feature vectors from each word in the text of the proprietary training set for each category to construct unit representations of knowledge units. Cluster the unit representations of each category of knowledge units to obtain multiple clusters. Determine representative prototype representations and extracted unit representations from each cluster. Use the original samples corresponding to the extracted unit representations to construct the core set for that category. S4. Execute the review phase training, including: obtaining a mixed training set consisting of a proprietary training set for the newly added category and a core set for each category, and using the preset total loss function, prototype representation, and mixed training set training to update the parameters of each expert network and encoding module.

2. The method according to claim 1, characterized in that For each category, the methods for determining representative prototype representations and extracting unit representations from each cluster include: Obtaining a unit representation of each recognition unit belonging to the category; The unit representations of this category are clustered using a preset clustering algorithm to obtain multiple clusters under this category; Using the centroid of each cluster under the category as the prototype representation of the cluster, multiple prototype representations under the category are obtained; An equal proportion of sampling is performed from each cluster under the category to obtain the extracted unit representation.

3. The method according to claim 2, characterized in that The binary cross entropy loss function in the preview stage is: Where i represents the number of the first word in the recognition unit in the input text, j represents the number of the second word in the recognition unit in the input text, and n represents the total number of words in the input text. Represents recognition unit s ij In the new label value of category k, p(k|s ij ) represents the k-th expert network for the recognition unit s ij Output prediction value, prediction value in the preview phase Represents knowledge units ij , sigmoid(·) represents the sigmoid function.

4. The method according to claim 3, characterized in that During the review phase, for the newly created expert network, the input text comes from a proprietary training set, and the preset total loss function includes the binary cross entropy loss function used to guide the learning of the newly created expert network: Among them, p′(k|s ij ) represents the kth expert network set in the review phase to identify the unit s ij The predicted value of the output, where: r ij =W6·ReLU(W5·e ij +b1)+b2 r k =W6·ReLU(W5·p k +b1)+b2 Among them, r ij Represents the unit representation of the recognition unit using the projection network ij The vector obtained by projection, r k Indicates that the prototype representation p is represented by the projection network k The vector obtained by projection, W5 represents the weight parameter of the first layer of the projection network, b1 represents the bias parameter of the first layer of the projection network, W6 represents the weight parameter of the second layer of the projection network, b2 represents the bias parameter of the second layer of the projection network, ReLU represents the ReLU activation function, Indicates e ij The prototypes of the same category and closest distance are represented by: Indicates e ij The shortest distance to the prototype representation of the same category, Indicates e ij The shortest distance between the prototype representation of category l, exp(·) represents the exponential function, and K represents the total number of categories.

5. The method according to claim 4, characterized in that For the expert network of non-new categories, the input text comes from the core set corresponding to the expert network. The distillation loss function in the review stage is: in, Indicates the soft distillation tag that needs to be imitated. is the predicted value output by the model after the previous class incremental learning, p(k|s ij ) represents the k-th expert network corresponding to the non-new category for the recognition unit s in the review phase ij The predicted value of the output, 6. The method according to claim 5, characterized in that The total loss function includes the weighted sum of the binary cross entropy loss function, the distillation loss function, and the contrast loss function. According to the total loss function, the parameters of each expert network, the encoding network, and the projection network are optimized to reduce the total loss. Among them, the contrast loss function is: Where N represents the total number of recognition units and τ1 represents the preset temperature hyperparameter.

7. The method according to claim 6, characterized in that The knowledge acquisition task is a named entity recognition task or a relationship extraction task, wherein the recognition unit under the named entity recognition task is the word-gram at the head and the word-gram at the tail of a paragraph in the input text, and the recognition unit under the relationship extraction task is the word-gram corresponding to two entities of the relationship category to be identified in the input text.

8. A quasi-incremental knowledge acquisition method based on a prototype network gating mechanism, characterized in that: include: Get the text to be predicted; Inputting the text to be predicted into the encoding module trained according to the method of any one of claims 1 to 7, obtaining the feature vector of each word unit, and constructing the unit representation of each knowledge unit using the feature vector of each word unit; The preset prototype network gating mechanism is used to determine the category to which each knowledge unit belongs, including: Obtain the similarity between the unit representation of each knowledge unit and the prototype representation of each category, select the partially matched expert network based on the similarity to predict the knowledge unit, and obtain the prediction value of whether the selected expert network belongs to the corresponding category of the expert network; For overlapping knowledge units, only the prediction value of the knowledge unit with the highest prediction value is retained, and when the highest prediction value is greater than or equal to a preset threshold, the retained knowledge unit is judged to belong to the category corresponding to the expert network that outputs the highest prediction value; If the predicted values ​​of a knowledge unit in all categories are less than the preset threshold, the knowledge unit does not belong to the category corresponding to any expert network.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.

Images