A method and system for continuous learning of text intent understanding

Abstract

The application provides a continuous learning method and system for text intention understanding, and the method comprises the following steps: when a new model needs to be trained, training data is acquired; wherein the training data comprises newly-added labeled data and part of historical labeled data that has been adopted by an original model; training batch data is constructed according to the newly-added labeled data and the part of historical labeled data that has been adopted by the original model; and the new model is trained according to the training batch data and a distribution constraint between the original model and the new model, so that a new version of the model is obtained; thereby, continuous learning of text intention is performed by using a small amount of historical labeled data in combination with newly-added labeled data; and the application focuses on optimizing the catastrophic forgetting problem in text intention understanding, so that the training time and training cost are reduced while the intention classification effect is ensured.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of natural language processing, and more particularly relates to a continuous learning method and system for text intention understanding. BACKGROUND

[0002] With the development of deep learning and natural language processing technology, more and more natural language processing services are provided for users on a cloud platform. In the field of human-computer interaction, such as intelligent sound boxes, smart televisions and other intelligent products, the cloud provides intention understanding services in various vertical fields (music, navigation, weather, etc.). With the expansion of the scope of text intention understanding, how the cloud intention understanding service can complete the expansion of new intentions more quickly with less training cost becomes an increasingly important problem.

[0003] A common processing scheme is to annotate more corpus of new intention types, and then retrain the model after merging all historical annotation data to realize the new intention category. However, this training method depends on the full amount of historical data, and the training time is long and the training cost is high. There are some attempts on continuous learning in the field of computer vision, but there is almost no related work in the academic field of NLP. The continuous learning system of text intention understanding has important value in actual industrial systems. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a continuous learning method and system for text intention understanding, which can reduce the training time and cost while ensuring the intention classification effect by using a small amount of historical annotation data in combination with new annotation data to continuously learn and optimize the catastrophic forgetting problem in text intention understanding.

[0005] The first aspect of the present application discloses a continuous learning method for text intention understanding, comprising:

[0006] When a new model needs to be trained, training data is obtained; wherein the training data includes new annotation data and part of historical annotation data already used by the original model;

[0007] According to the new annotation data and the part of historical annotation data already used by the original model, training batch data is constructed;

[0008] According to the training batch data and the distribution constraint between the original model and the new model, the new model is trained to obtain a new version of the model.

[0009] Optionally, in the continuous learning method for text intention understanding, the training data is obtained, comprising:

[0010] Filtering historical data to obtain part of the historical annotation data; wherein, the historical data is the annotation data already adopted by the original model;

[0011] The newly added annotation data is expanded through a data enhancement method.

[0012] Optionally, in the continuous learning method for text intent understanding, a training batch data is constructed according to the newly added annotation data and the part of the historical annotation data already adopted by the original model, comprising:

[0013] The newly added annotation data after enhancement is taken as a first data set, and the part of the historical annotation data already adopted by the original model is taken as a second data set; and the first data set and the second data set are fused to obtain a training set;

[0014] According to the training set, batch data required for multiple training is obtained respectively; wherein, the batch data required for each training is a corpus of a preset data selected from the first data set and the second data set respectively.

[0015] Optionally, in the continuous learning method for text intent understanding, according to the training batch data and the distribution constraint between the original model and the new model, the new model is trained to obtain a new version model, comprising:

[0016] The data from the historical annotation data in the training batch data is inferred by the original model to record the feature distribution result of the original model;

[0017] The data of the historical annotation data is input into the new model, and the result of the new model is constrained to have the same performance as the result of the original model through a distribution loss function;

[0018] The data from the newly added annotation data in the training batch data is iteratively trained by the new model; and,

[0019] The distance between the data of the newly added annotation data annotated as a new category and the data of an old category is greater than a preset value; wherein, the newly added category in the newly added annotation data is the new category, and the old category is the category already adopted by the original model.

[0020] Optionally, in the continuous learning method for text intent understanding, the iterative training adopts a classification loss function constraint; and / or, the distance between the new category and the old category is controlled to be greater than a preset value through a bias loss function.

[0021] Optionally, in the continuous learning method for text intent understanding, the distance between the data of the newly added annotation data annotated as a new category and the data of an old category is greater than a preset value, comprising:

[0022] The center of each classification category represents different;

[0023] The center of each classification category is set or updated so that the distance between the new classification and the center of the old category is greater than the preset distance.

[0024] Optionally, in the continuous learning method for text intention understanding, the overall constraint loss of the model is trained according to the training batch data:

[0025] Loss = α1*L distri + α2*L classify + α3*L margin

[0026]

[0027]

[0028] α i represents the weight of the i-th Loss; L distri is a distribution loss function; L classify is a classification loss function; L margin bias loss function; δ i is the ratio of the loss value obtained by the i-th Loss in the last two training batches; δ j is the ratio of the loss value obtained by the j-th Loss in the last two training batches; L i is the i-th Loss value; ΔL i is the calculation method of δ i ; S is the batch count of model training; Loss i is the true value of L i of the S-th training batch.

[0029] The second aspect of the present application discloses a continuous learning system for text intention understanding, comprising:

[0030] A data acquisition unit is configured to acquire training data when a new model needs to be trained; wherein the training data includes newly added labeled data and part of historical labeled data already adopted by the original model;

[0031] A construction unit is configured to construct training batch data according to the newly added labeled data and the part of historical labeled data already adopted by the original model;

[0032] A training unit is configured to train the new model according to the training batch data and the distribution constraint between the original model and the new model, and obtain a new version of the model.

[0033] Optionally, in the continuous learning system for text intention understanding, when the data acquisition unit is used to acquire training data, the data acquisition unit is specifically used for:

[0034] filtering historical data, wherein the historical data is labeled data already adopted by the original model;

[0035] performing corpus expansion on the newly added labeled data by using a data enhancement method.

[0036] Optionally, in the continuous learning system for text intention understanding, when the construction unit is used to construct training batch data according to the newly added labeled data and the part of historical labeled data already adopted by the original model, the construction unit is specifically used for:

[0037] using the newly added labeled data after enhancement as a first data set, using the part of historical labeled data already adopted by the original model as a second data set, and fusing the first data set and the second data set to obtain a training set;

[0038] acquiring batch data required for multiple times of training respectively according to the training set, wherein the batch data required for each time of training is corpus of pre-set data selected from the first data set and the second data set.

[0039] As can be seen from the above technical solution, the continuous learning method for text intention understanding provided by the present application includes: when a new model needs to be trained, acquiring training data; wherein the training data includes newly added labeled data and part of historical labeled data already adopted by the original model; constructing training batch data according to the newly added labeled data and the part of historical labeled data already adopted by the original model; training the new model according to the training batch data and the distribution constraint between the original model and the new model to obtain a new version of the model; thereby, the continuous learning of text intention is performed by using a small amount of historical labeled data in combination with newly added labeled data; compared with using full-amount historical labeled data, using a small amount of historical labeled data for model iteration will encounter a “catastrophic forgetting problem” of a deep learning model; the performance on an old task will usually decrease significantly when training on a new task; that is, the present application focuses on optimizing the catastrophic forgetting problem in text intention understanding, and reduces training time and training cost while ensuring the intention classification effect. BRIEF DESCRIPTION OF DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0041] Figure 1 is a flow chart of a continuous learning method for text intent understanding provided by an embodiment of the present application;

[0042] Figure 2 is a flow chart of another continuous learning method for text intent understanding provided by an embodiment of the present application;

[0043] Figure 3 is a flow chart of another continuous learning method for text intent understanding provided by an embodiment of the present application;

[0044] Figure 4 is a logic diagram of a continuous learning method for text intent understanding provided by an embodiment of the present application;

[0045] Figure 5 is a diagram of batch data involved in a continuous learning method for text intent understanding provided by an embodiment of the present application;

[0046] Figure 6 is a diagram of a model training framework and loss relationship involved in a continuous learning method for text intent understanding provided by an embodiment of the present application;

[0047] Figure 7 is a logic diagram of classification centroid calculation and update involved in a continuous learning method for text intent understanding provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0049] In the present application, the terms “comprising”, “containing” or any other variants thereof are intended to cover non-exclusive containing, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitations, the element defined by the statement “comprising a” does not exclude the presence of another identical element in the process, method, article or equipment including the element.

[0050] The embodiment of the application discloses a continuous learning method for text intention understanding, and is used for solving the problems of dependence on historical full data, long training time and high training cost in the prior art.

[0051] Referring to Figure 1 The continuous learning method for text intention understanding comprises the following steps.

[0052] In S101, training data is acquired when a new model needs to be trained.

[0053] The training data comprises newly-added annotation data and part of historical annotation data that has been used by an original model.

[0054] It should be noted that, generally, the historical data refers to the annotation data that has been used in the previous version of the model.

[0055] The newly-added annotation data refers to some newly-generated annotation data to be trained, and is used for iteration of a new version of the model.

[0056] Generally, the annotation data adopts a general scheme of classification annotation, and one piece of text corresponds to one valid label. For example, "text: open the window, label: open". The specific annotation process is not described here, and can be determined according to actual conditions, and is within the protection scope of the application.

[0057] In order to enable the new iteration model to better support the results of the original classification, a small amount of historical data and new data need to be mixed together for joint training. It is difficult to guarantee the effect through verification without using the historical data, and the catastrophic forgetting problem of the model cannot be well solved, therefore, part of the historical annotation data and the newly-added annotation data are used as training data together, the catastrophic forgetting problem in text intention understanding is optimized, the training time and the training cost are reduced under the condition of guaranteeing the intention classification effect.

[0058] The small amount generally refers to about 10% to 20% of the original full data. Of course, it can also be other numerical values, which are not limited here, and can be determined according to actual conditions, and are within the protection scope of the application.

[0059] In S102, training batch data is constructed according to the newly-added annotation data and part of the historical annotation data that has been used by the original model.

[0060] In S103, the new model is trained according to the training batch data and the distribution constraint between the original model and the new model, and a new version of the model is obtained.

[0061] It should be noted that by controlling the distribution constraint between the historical model and the new model, the key results of the historical model can be fitted in the process of training the new model, so as to ensure that the historical effect will not be forgotten, and at the same time the new class effect can be trained to enhance. The historical model described here is the original model described above, and the new model is the new model described above.

[0062] It should be noted that there are some attempts in the field of computer vision for continuous learning, such as the related papers A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks, A comprehensive, application-oriented study of catastrophic forgetting in DNNs; but there is almost no related work in the academic field of NLP, and the continuous learning system of text intent understanding has important value in the actual industrial system.

[0063] As shown in Figure 4 , which shows the logical diagram of the continuous learning method of text intent understanding; that is, the historical data is filtered on demand to obtain data set A; and the newly labeled data is augmented to obtain data set B; data set A and data set B are fused to obtain training data set C, and the constraint between the original model and the new model is obtained according to the training data set C. The new version model.

[0064] In this embodiment, when it is necessary to train a new model, training data is obtained; wherein the training data includes newly labeled data and part of the historical labeled data already used by the original model; according to the newly labeled data and part of the historical labeled data already used by the original model, training batch data is constructed; according to the training batch data and the distribution constraint between the original model and the new model, the new model is trained to obtain a new version model; thereby, by using a small amount of historical labeled data, combined with newly labeled data, continuous learning of text intent is performed; compared with using full historical labeled data, using a small amount of historical labeled data for model iteration will encounter the "catastrophic forgetting problem" of deep learning model; the performance on the old task usually decreases significantly when training on the new task; that is, the present application focuses on optimizing the catastrophic forgetting problem in text intent understanding, and reduces the training time and training cost while ensuring the intent classification effect.

[0065] In practical applications, referring to Figure 2 , the step S101 involves obtaining training data, including:

[0066] S201, screening the historical data to obtain part of the historical labeled data.

[0067] The historical data is labeled data that has been adopted by the original model.

[0068] In actual application, we need to select a small amount of valuable training data from the large amount of historical labeled data through some methods, and use the screened data as the data set A.

[0069] The data set A is combined with the incremental labeled data to form a new training data set C.

[0070] It should be noted that the screening of historical data needs the category distribution of historical data, more valuable edge data and other factors.

[0071] The specific screening method can be: screening data by business function unit; screening edge data according to the effect of the original model. Business function refers to the smallest functional unit in the business scenario of the intent understanding model application, such as helping me open the window, close the window, and the corresponding function label is open, close. Then open and close are business functions.

[0072] The screening method can be one of the two ways, or a mixture of the two ways, without distinction and combination limitation according to the actual situation, which will not be repeated here, and can be determined according to the actual situation, which is within the protection scope of the present application.

[0073] Edge data refers to some data on the original data whose classification result score is on the edge of the expected result. These data are a subset of the original data, and have more learning value for the model.

[0074] S202, expanding the corpus by data augmentation method for the new labeled data.

[0075] The corpus is expanded by data augmentation method for the new labeled data, realizing data augmentation of the new labeled data.

[0076] Specifically, the data augmentation method can be a general NLP (Natural language processing) processing method, which includes synonym replacement, translation back-translation and the like. The specific processing process will not be repeated here, and can be determined according to the actual situation, which is within the protection scope of the present application.

[0077] In addition, the data augmentation method can also be other processing methods, which will not be repeated here, and can be determined according to the actual situation, which is within the protection scope of the present application.

[0078] In practical applications, the above step S102, according to the newly added annotation data and the part of historical annotation data already adopted by the original model, constructs training batch data, including:

[0079] (1) The newly added annotation data after enhancement is taken as a first data set, and the part of historical annotation data already adopted by the original model is taken as a second data set; and the first data set and the second data set are fused to obtain a training set.

[0080] That is, the new training data set is composed of a small amount of historical data selected on demand and the newly added annotation data after data enhancement.

[0081] Specifically, the training set C is obtained by fusing the set A selected from the historical data and the set B after data enhancement of the newly added annotation data.

[0082] (2) According to the training set, batch data required for multiple training is obtained respectively.

[0083] Among them, the batch data required for each training is the corpus of the preset data selected from the first data set and the second data set.

[0084] Specifically, the batch data required for single training is composed as shown in Figure 5 . In each training process, we need to randomly select a part of data corpus from the set A and the set B. The proportion of these different set corpora is usually 1:1, of course, it can also be other proportions, which will not be described one by one here, and can be determined according to the actual situation, which is within the protection scope of the present application.

[0085] In addition, the selected corpus covers more different intent categories as much as possible. The size of the batch data can be 1024 or 4096, etc. according to actual needs; this size is mainly limited by the video memory of the training machine; in Figure 5 , old refers to the original data amount, new refers to the incremental data amount; open and close refer to the label value of the intent annotation.

[0086] Figure 5 is the composition of each batch data in the training step, wherein the original data is the training corpus of the last iteration, and the incremental data is the incremental training corpus newly added this time. In the incremental training corpus, there can be original defined intent categories (open), and there can be newly added intent categories (close).

[0087] In practical applications, referring to Figure 3 , step S103, according to the training batch data and the distribution constraint between the original model and the new model, the new model is trained to obtain a new version of the model, including:

[0088] S301. Take the data from the historical labeled data in the training batch data, and record the feature distribution results of the original model through inference.

[0089] In other words, firstly, the data from source dataset A in the batch data is inferred using the historical model, and the feature distribution results of the historical model are recorded. The batch data is the training set C.

[0090] S302. Input the historical labeled data into the new model, and use the distribution loss function to constrain the results of the new model to have the same performance as the original model.

[0091] Data from dataset A is processed through a new model, using a distribution loss function (such as...). Figure 6 The Loss-distribution constraint (DSC) indicates that the results of the new model are consistent with those of the historical model, meaning that the distribution of the features of these data is consistent in both the new and old models.

[0092] The distribution loss function Loss-distri can use common cross-entropy loss or appropriate distillation loss (Loss); these will not be elaborated here, and can be chosen according to the actual situation, all of which are within the scope of protection of this application.

[0093] S303. Take the newly labeled data from the training batch data and iteratively train it using the new model.

[0094] The data from source dataset B in the batch is used to iteratively train the new model. Here, a classification loss function (such as...) is used. Figure 6 As shown in the Loss-classify constraint, the classification results in dataset B are the same as the manually labeled results in dataset B.

[0095] In practical applications, the classification loss function Loss-classify can typically use cross-entropy, exponential loss, etc.; these will not be elaborated here, and can be chosen according to the actual situation, all of which are within the scope of protection of this application.

[0096] S304. Control the spacing between newly added labeled data and data in the old category to be greater than the preset value.

[0097] Among them, the newly added categories in the newly labeled data are new categories, and the old categories are categories that have been used in the original model.

[0098] In dataset B, the distance between data labeled as belonging to the new category and those in the old category is greater than a preset value; that is, the distance between them is significant. To ensure this, a new bias loss function (such as...) needs to be introduced. Figure 6The shown Loss-margin).

[0099] The bias loss function Loss-margin can generally use Hinge Loss, Margin loss, etc.; here, it is not necessary to repeat, and it can be determined according to the actual situation, which is within the protection scope of the present application.

[0100] That is, the iterative training is constrained by the classification loss function; and / or, the distance between the new class and the old class is controlled to be greater than a preset value by the bias loss function.

[0101] It should be noted that the above steps S301-S304 can be executed simultaneously or in a specific order, which is not repeated here, and can be determined according to the actual situation, which is within the protection scope of the present application.

[0102] It should be noted that the text intent understanding model structure is as shown in Figure 6 The text content is modeled by word or word or mixed embedding Embedding, and the hidden layer output is obtained through the feature extraction layer, and then the hidden layer output is connected to the classification layer to obtain the text intent classification result. The actual training steps are shown in Figure 6 .

[0103] Figure 3 is the framework logic of model continuous learning. The left side except Q1 part is the inference stage of the original model. The original data needs to request the original model at the same time, and record the related deep learning prediction results, which are used for the new model to fit the effect of the original data on the original model, and used for the new model to "maintain old memory".

[0104] In practical applications, the combination of the above three types of constraint conditions can make the model not be forgotten on the old data, and get new training results on the new data.

[0105] In practical applications, the distance between the data labeled as a new class and the old class is controlled to be greater than a preset value, including:

[0106] The center of each classification category represents different; set or update the center of each classification category, so that the distance between the new classification and the center of the old classification is greater than the preset distance.

[0107] How to ensure that the data labeled as a new class in set B is far away from the old class. The center (centroid) of each classification category can be recorded, and the distance from the classification center point is calculated to make the classification result far away from the center of other categories.

[0108] Specifically, the representation and update of the centroid of all classifications need to be calculated by a specific method. The specific method is as follows:Figure 7 The original model is recorded and stored in the representation of each classification result. For one category, the centroid representation of this category is calculated by the hidden layer features of multiple use cases belonging to this category through some centroid calculation methods, such as common average calculation method.

[0109] If there are N categories, there will be N centroid representations of categories, which need to be stored for subsequent model training process; the centroid of the old category is obtained in the historical model, and the centroid of the new category is initially a 0 vector, which is obtained after the first batch training. After the training process is started, the centroids of all categories will be calculated and updated, and the timing of the update is after the end of each batch forward inference, and the update is restarted.

[0110] As shown in Figure 7 For the old category ID1 (unique identification), the original centroid has been calculated and can be directly obtained from the storage. After the completion of a round of batch forward inference calculation, the features of a number of corpus belonging to category ID1 are recorded, and then the original centroid vector and the feature vector of the corpus of ID1 category are put together to recalculate and update the new centroid. For the category IDn-2 in the figure, there is no such category in the historical centroid data, so the initial centroid vector is 0. After the first batch forward calculation, the corpus belonging to the n-2 category is combined in a set, and the centroid representation of the new category is obtained through the centroid calculation algorithm. This process is repeated continuously with the development of batch training process. When the model training is completed, the centroid vectors of all categories will be stored for the next model iteration.

[0111] It should be noted that the above model training steps involve the joint action of three different constraint conditions, and how to use the three effective loss functions together to get the best model effect is very critical. The present scheme proposes a method of dynamically updating the weight adjustment of the three constraint losses (Loss) to maximize the overall loss (Loss) to the model.

[0112] That is, in actual application, the overall constraint loss of the model trained according to the training batch data is:

[0113] Loss=α1*L distri +α2*L classify +α3*L margin

[0114]

[0115]

[0116] αi represents the weight of the ith Loss; L distri is a distribution loss function; L classify is a classification loss function; L margin is a bias loss function; δ i is the ratio of the loss values of the previous two training batches of the ith Loss; δ j is the ratio of the loss values of the previous two training batches of the jth Loss; L i is the ith Loss value; ΔL i is δ i is calculated; S is the batch count of model training; Loss i is the true value of the L i of the Sth training batch.

[0117] It should be noted that the weight of each is calculated according to the ratio of the previous two iteration loss (Loss) values and the loss (Loss) value of the previous step, and is obtained by normalizing operation. If there is no corresponding loss (Loss) value initially, the weight is 0. The weight information is recalculated after each replacement calculation.

[0118] Another embodiment of the present application provides a continuous learning system for text intent understanding.

[0119] The continuous learning system for text intent understanding comprises:

[0120] A data acquisition unit is configured to acquire training data when a new model needs to be trained; wherein the training data comprises newly added labeled data and part of historical labeled data already adopted by an original model.

[0121] A construction unit is configured to construct training batch data according to the newly added labeled data and the part of historical labeled data already adopted by the original model.

[0122] A training unit is configured to train a new model according to the training batch data and a distribution constraint between the original model and the new model, to obtain a new version of the model.

[0123] In actual application, when the data acquisition unit is used to acquire training data, it is specifically used to:

[0124] filter historical data; wherein the historical data is labeled data already adopted by the original model; and newly added labeled data is corpus augmented by a data enhancement method.

[0125] In actual application, when the construction unit is used to construct training batch data according to the newly added labeled data and the part of historical labeled data already adopted by the original model, it is specifically used to:

[0126] The newly added labeled data is enhanced as a first data set, and part of the historical labeled data already used by the original model is taken as a second data set; and the first data set and the second data set are fused to obtain a training set.

[0127] According to the training set, batch data required for multiple training is obtained respectively; wherein, the batch data required for each training is corpus of pre-set data selected from the first data set and the second data set respectively.

[0128] The working process and principle of each unit are described in detail in the continuous learning method for text intent understanding provided in the above embodiments, which will not be repeated here, and can be determined according to the actual situation, and are within the protection scope of the present application.

[0129] In the present embodiment, the data acquisition unit acquires training data when a new model needs to be trained; wherein, the training data includes newly added labeled data and part of the historical labeled data already used by the original model; the construction unit constructs training batch data according to the newly added labeled data and part of the historical labeled data already used by the original model; the training unit trains the new model according to the training batch data and the distribution constraint between the original model and the new model, to obtain a new version of the model; thereby, the continuous learning of the text intent is performed by using a small amount of historical labeled data in combination with newly added labeled data; compared with using full amount of historical labeled data, using a small amount of historical labeled data for model iteration will encounter the "catastrophic forgetting problem" of deep learning model; the performance on the old task will usually decrease significantly when training on the new task; that is, the present application focuses on optimizing the catastrophic forgetting problem in text intent understanding, and reduces the training time and training cost while ensuring the intent classification effect.

[0130] The features described in each of the embodiments in the present specification can be replaced or combined with each other, and the same or similar parts between each embodiment can be referred to each other. Each embodiment focuses on the difference from other embodiments. Especially, for the system or system embodiment, since it is basically similar to the method embodiment, it is described more simply, and the related parts can be referred to the part of the method embodiment. The above described system and system embodiment are only illustrative, wherein the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to the actual needs, part or all of the modules can be selected to achieve the purpose of the present embodiment. Those skilled in the art can understand and implement without creative labor.

[0131] Those skilled in the art will further realize that the mechanisms of the various examples described herein are capable of being implemented using any number of combinations of the described features. Accordingly, these examples are not limited to the mechanisms described herein, but rather, the intent is to cover all modifications and alternatives equivalent thereto. The preceding description of the examples is illustrative, and not restrictive. Many other examples will be apparent to those of skill in the art upon reviewing the above description. The scope of the examples should, therefore, be determined not with reference to the above description, but instead should be given to the appended claims, along with their full scope of equivalents.

[0132] The above description of disclosed examples allows one of ordinary skill in the art to make and use the application. Modifications of those examples will be obvious to those of ordinary skill in the art, and the generic principles defined herein can be applied to other examples without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the examples shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for continuous learning of text intent understanding, characterized in that, The method comprises the following steps: When a new model needs to be trained, training data is acquired; wherein the training data is text data; the training data comprises newly added annotation data and part of historical annotation data already adopted by an original model; Training batch data is constructed according to the newly added annotation data and the part of historical annotation data already adopted by the original model; The new model is trained according to the training batch data and a distribution constraint between the original model and the new model, to obtain a new version model, which comprises: the data from the historical annotation data in the training batch data is input into the original model for reasoning, and the feature distribution result of the original model is recorded; the data of the historical annotation data is input into the new model, and a distribution loss function is used to constrain the result of the new model to have the same performance as the result of the original model; the data from the newly added annotation data in the training batch data is input into the new model for iterative training; and the distance between the data of the newly added annotation data annotated as a new category and the data of an old category is greater than a preset value; wherein the new category in the newly added annotation data is the new category, and the old category is the category already adopted by the original model.

2. The method of claim 1, wherein, The training data is acquired, comprising: Part of historical annotation data is obtained by screening historical data; wherein the historical data is annotation data already adopted by an original model; The newly added annotation data is expanded by a data enhancement method. 3.The method of claim 2, wherein, The training batch data is constructed according to the newly added annotation data and the part of historical annotation data already adopted by the original model, comprising: The newly added annotation data after enhancement is taken as a first data set, and the part of historical annotation data already adopted by the original model is taken as a second data set; and the first data set and the second data set are fused to obtain a training set; Batch data required for multiple times of training is acquired according to the training set; wherein the batch data required for each time of training is a preset data corpus selected from the first data set and the second data set respectively.

4. The method of claim 1, wherein, The iterative training adopts a classification loss function constraint; and / or the distance between the new category and the old category is greater than a preset value by a bias loss function.

5. The method of claim 1, wherein, The distance between the data of the newly added annotation data annotated as a new category and the data of an old category is greater than a preset value, comprising: The center of each classification category is different; The center of each classification category is set or updated, so that the distance between the new category and the old category is greater than a preset distance.

6. The method of claim 4, wherein, The overall constraint loss for training the model according to the training batch data is: ； weight of the ith Loss; is a distribution loss function; is a classification loss function; is a bias loss function; is a ratio of loss values of the ith Loss in the last two training batches; is a ratio of loss values of the jth Loss in the last two training batches; is the ith Loss value; is a calculation method of S is a batch count of model training; is a true value of the Sth training batch of is an overall constraint loss; the weight of the ith Loss is a weight value of the ith overall constraint loss; the ith Loss value is a value of the ith overall constraint loss.

7. A system for continual learning of text intent understanding, the system comprising: The method comprises the following steps: A data acquisition unit is configured to acquire training data when a new model needs to be trained; wherein the training data is text data; the training data comprises newly added annotation data and part of historical annotation data already adopted by an original model; A construction unit is configured to construct training batch data according to the newly added annotation data and the part of historical annotation data already adopted by the original model; The training unit is configured to train the new model according to the training batch data, and a distribution constraint between the original model and the new model, to obtain a new version of the model, including: inputting data from the historical annotation data in the training batch data into the original model to record a feature distribution result of the original model; inputting data of the historical annotation data into the new model to constrain a result of the new model to have the same performance as a result of the original model by using a distribution loss function; inputting data from new annotation data in the training batch data into the new model for iterative training; and controlling a distance between data annotated as a new class and data annotated as an old class in the new annotation data to be greater than a preset value; wherein the new class is a new class in the new annotation data, and the old class is a class already used by the original model.

8. The system of claim 7, wherein, The data acquisition unit is configured to acquire training data, specifically configured to: filter historical data; wherein the historical data is annotation data already used by the original model; perform corpus expansion on the new annotation data by using a data enhancement method.

9. The system of claim 8, wherein, The construction unit is configured to construct training batch data according to the new annotation data and part of the historical annotation data already used by the original model, specifically configured to: use the enhanced new annotation data as a first data set, use part of the historical annotation data already used by the original model as a second data set, and fuse the first data set and the second data set to obtain a training set; acquire batch data required for multiple times of training according to the training set; wherein the batch data required for each time of training is corpus of a preset amount of data selected from the first data set and the second data set.

Images