A method and system for incremental training and continuous deployment of text classification models

Abstract

The embodiment of the present invention discloses a method and system for incremental training and continuous deployment of a text classification model, the method comprising: generating feedback data according to text data to be classified input by a user; updating a full training data set, an incremental training data set and a test data set based on the text data to be classified, the feedback data and the imported annotation data according to the selected model training method; generating multiple first text classification models based on the above data sets according to pre-set model training rules; and selecting a first text classification model that meets the model deployment trigger condition from multiple first text classification models as a second text classification model for service deployment according to a pre-set model deployment trigger condition. The method and system reduce the cost of manual data annotation, ensure that the text classification model obtained according to the continuously updated training data can adapt to the needs of constantly changing business scenarios, simplify the operation of operation and maintenance personnel, and improve the operation and maintenance efficiency.

Description

Technical Field

[0001] The present invention relates to the field of data classification, and in particular to a method and system for incremental training and continuous deployment of a text classification model. Background Art

[0002] Text classification is a basic task in natural language processing and has a wide range of applications, such as spam filtering, news classification, and product code recognition. Currently, the best classification methods are deep learning models, such as fastText and CNN models. Such models rely on a large amount of labeled data to work properly, and generally the larger the amount of data, the better the effect. In actual production practice, there are the following problems: 1) Since data labeling consumes a lot of manpower, the amount of labeled data at one time may not be able to fully meet the model of the current business scenario. 2) Due to the continuous updating of business scenarios, the model trained based on previous labeled data cannot meet new business needs. 3) Feedback from users during the use of the model is ignored. 4) The incremental model training process requires manual participation, and the model cannot be trained in time. Summary of the invention

[0003] In order to solve the problems in the prior art of incremental training and deployment of text classification models, such as the limited amount of annotated data in the training data set at one time, which cannot meet the business scenarios, the original model caused by the update of the business scenarios cannot meet the new business needs, the feedback from users in the process of using the model is ignored, and the incremental training of the model requires manual participation, the embodiments of the present invention provide a method and system for incremental training and continuous deployment of a text classification model.

[0004] According to one aspect of an embodiment of the present invention, a method for incremental training and continuous deployment of a text classification model is provided, the method comprising:

[0005] Generate feedback data according to the text data to be classified input by the user;

[0006] According to the selected model training method, update the full training dataset, incremental training dataset, and test dataset based on the text data to be classified, feedback data, and imported annotation data;

[0007] Based on the incremental training data set, the full training data set and the test data set, and according to pre-set model training rules, a plurality of first text classification models are generated;

[0008] According to the preset model deployment trigger condition, a first text classification model that meets the model deployment trigger condition is selected from multiple first text classification models as the second text classification model for service deployment.

[0009] Optionally, in the above-mentioned method embodiments of the present invention, the method also includes detecting the availability of a second text classification model used for service deployment, and when the second text classification model is available, switching the external access traffic of the service to the second text classification model, and stopping the current text classification model used for service deployment; when the second text classification model is unavailable, generating a warning message.

[0010] Optionally, in the above-mentioned method embodiments of the present invention, generating feedback data according to the text data to be classified input by the user includes:

[0011] For the text data to be classified input by the user, the current text classification model deployed by the service is called to generate multiple classification results;

[0012] When the feedback data generation mode is passive feedback, one classification result is selected from the multiple classification results as the feedback data; when the feedback data generation mode is active feedback, the multiple classification results are discarded and a new classification result is generated as the feedback data.

[0013] Optionally, in the above-mentioned method embodiments of the present invention, according to the selected model training mode, updating the full training data set, the incremental training data set and the test data set based on the text data to be classified, the feedback data and the imported annotation data includes:

[0014] When the model training mode is incremental training, the text data to be classified, the feedback data and the first labeled data are added to the incremental training data set, wherein the imported labeled data includes the first labeled data and the second labeled data, and the initial value of the cardinality of the incremental training data set is 0;

[0015] When the model training mode is full training, the elements in the incremental training data set and the first labeled data are added to the full training data set according to the data addition trigger strategy, wherein the data addition trigger strategy includes timed trigger, quantitative trigger and manual trigger. The quantitative trigger means that data addition is performed after the cardinality of the incremental training data set reaches the set update trigger threshold. The initial value of the cardinality of the full training data set is 0.

[0016] When performing incremental training or full training, the second labeled data is added to the test data set, wherein the initial value of the cardinality of the test data set is 0.

[0017] Optionally, in the above-mentioned method embodiments of the present invention, based on the incremental training data set, the full training data set and the test data set, generating multiple first text classification models according to pre-set model training rules includes:

[0018] Determine the initial values ​​of the model parameters of the training text classification model according to the selected model training method;

[0019] According to the initial values ​​of the model parameters, a training data set corresponding to the model training method is used, and model training is performed according to the selected model training trigger method to generate multiple pending text classification models, wherein when the model training method is full training, the full training data set is used; when the model training method is incremental training, the incremental training data set is used; when the model training trigger method is manual triggering, the model training is manually triggered; when the model training trigger method is periodic triggering, the model training is triggered according to the set training period and training method; when the model training trigger method is indicator decline triggering, the model training is triggered when it is detected that the model performance indicator declines to the set training trigger threshold;

[0020] For each generated pending text classification model, a test data set is used to detect the model performance. When the model performance index value of the pending text classification model is greater than a preset performance threshold, the pending text classification model is determined to be the first text classification model. When the model performance index value of the pending text classification model is not greater than the preset performance threshold, the pending text classification model is discarded.

[0021] Optionally, in the above-mentioned method embodiments of the present invention, according to a preset model deployment trigger condition, selecting a first text classification model that meets the model deployment trigger condition from multiple first text classification models as a second text classification model for service deployment includes:

[0022] When the model deployment trigger condition is to query the first text classification model with the highest model performance index value within a time period, the model performance index values ​​of the multiple first text classification models are sorted from large to small;

[0023] The first text classification model ranked first is selected as the second text classification model for service deployment.

[0024] Optionally, in the above-mentioned method embodiments of the present invention, the method further comprises calculating user satisfaction according to user behavior data for the second text classification model used for service deployment, and determining the performance level of the second text classification model according to the correspondence between the user satisfaction interval and the model performance level, wherein:

[0025] The formula for calculating user satisfaction S based on user behavior data is:

[0026] S = (number of active feedbacks + number of passive feedbacks) / number of times the second text classification model is called.

[0027] According to another aspect of an embodiment of the present invention, a system for incremental training and continuous deployment of a text classification model is provided, the system comprising:

[0028] A feedback data module, used to generate feedback data according to the text data to be classified input by the user;

[0029] A data processing module is used to update the full training data set, the incremental training data set and the test data set based on the text data to be classified, the feedback data and the imported annotation data according to the selected model training method;

[0030] A model training module, used to generate a plurality of first text classification models based on an incremental training data set, a full training data set and a test data set according to a preset model training rule;

[0031] The model deployment module is used to select a first text classification model that meets the model deployment trigger condition from multiple first text classification models as a second text classification model for service deployment according to a preset model deployment trigger condition.

[0032] Optionally, in the above-mentioned device embodiments of the present invention, the model deployment module also detects the availability of a second text classification model used for service deployment. When the second text classification model is available, the external access traffic of the service is switched to the second text classification model, and the current text classification model used for service deployment is stopped; when the second text classification model is unavailable, a warning message is generated.

[0033] Optionally, in each of the above-mentioned device embodiments of the present invention, the feedback data module generates feedback data according to the text data to be classified input by the user, including:

[0034] For the text data to be classified input by the user, the current text classification model deployed by the service is called to generate multiple classification results;

[0035] When the feedback data generation mode is passive feedback, one classification result is selected from the multiple classification results as the feedback data; when the feedback data generation mode is active feedback, the multiple classification results are discarded and a new classification result is generated as the feedback data.

[0036] Optionally, in each of the above-mentioned device embodiments of the present invention, the data processing module updates the full training data set, the incremental training data set and the test data set based on the text data to be classified, the feedback data and the imported annotation data according to the selected model training method, including:

[0037] When the model training mode is incremental training, the text data to be classified, the feedback data and the first labeled data are added to the incremental training data set, wherein the imported labeled data includes the first labeled data and the second labeled data, and the initial value of the cardinality of the incremental training data set is 0;

[0038] When the model training mode is full training, the elements in the incremental training data set and the first labeled data are added to the full training data set according to the data addition trigger strategy, wherein the data addition trigger strategy includes timed trigger, quantitative trigger and manual trigger. The quantitative trigger means that data addition is performed after the cardinality of the incremental training data set reaches the set update trigger threshold. The initial value of the cardinality of the full training data set is 0.

[0039] When performing incremental training or full training, the second labeled data is added to the test data set, wherein the initial value of the cardinality of the test data set is 0.

[0040] Optionally, in each of the above-mentioned device embodiments of the present invention, the model training module generates a plurality of first text classification models based on the incremental training data set, the full training data set and the test data set according to a preset model training rule, including:

[0041] Determine the initial values ​​of the model parameters of the training text classification model according to the selected model training method;

[0042] According to the initial values ​​of the model parameters, a training data set corresponding to the model training method is used, and model training is performed according to the selected model training trigger method to generate multiple pending text classification models, wherein when the model training method is full training, the full training data set is used; when the model training method is incremental training, the incremental training data set is used; when the model training trigger method is manual triggering, the model training is manually triggered; when the model training trigger method is periodic triggering, the model training is triggered according to the set training period and training method; when the model training trigger method is indicator decline triggering, the model training is triggered when it is detected that the model performance indicator declines to the set training trigger threshold;

[0043] For each generated pending text classification model, a test data set is used to detect the model performance. When the model performance index value of the pending text classification model is greater than a preset performance threshold, the pending text classification model is determined to be the first text classification model. When the model performance index value of the pending text classification model is not greater than the preset performance threshold, the pending text classification model is discarded.

[0044] Optionally, in each of the above-mentioned device embodiments of the present invention, the service deployment module selects, according to a preset model deployment trigger condition, a first text classification model that meets the model deployment trigger condition from a plurality of first text classification models as a second text classification model for service deployment, including:

[0045] When the model deployment trigger condition is to query the first text classification model with the highest model performance index value within a time period, the model performance index values ​​of the multiple first text classification models are sorted from large to small;

[0046] The first text classification model ranked first is selected as the second text classification model for service deployment.

[0047] Optionally, in each of the above-mentioned device embodiments of the present invention, the system further includes a satisfaction evaluation module, which calculates user satisfaction according to user behavior data for the second text classification model used for service deployment, and determines the performance level of the second text classification model according to the corresponding relationship between the user satisfaction interval and the model performance level, wherein:

[0048] The formula for calculating user satisfaction S based on user behavior data is:

[0049] S = (number of active feedbacks + number of passive feedbacks) / number of times the second text classification model is called.

[0050] Based on the method and system for incremental training and continuous deployment of text classification models provided by the above-mentioned embodiments of the present invention, the method includes: generating feedback data according to text data to be classified input by a user; updating a full training data set, an incremental training data set and a test data set based on the text data to be classified, the feedback data and the imported annotation data according to the selected model training method; generating multiple first text classification models based on the incremental training data set, the full training data set and the test data set according to pre-set model training rules; and selecting a first text classification model that meets the model deployment trigger condition from multiple first text classification models according to a pre-set model deployment trigger condition as a second text classification model for service deployment. The beneficial effects of the method and system include: 1) designing a user feedback mechanism, taking the user feedback data as one of the data sources of the incremental training data set, thereby reducing the cost of manually annotating data, and calculating user satisfaction based on user feedback as a reference indicator for evaluating the performance of the classification model used for service deployment after training, making the evaluation of model performance more comprehensive; 2) designing a method for generating model training data, taking the text data and feedback data uploaded by the user, and the imported annotated data as the source of the incremental training data set for incremental training, and when the model training is full training, taking the incremental training data set and the newly imported annotated data together as the source of the full training data set, thereby realizing the continuous updating of the model training data, and at the same time At the same time, the labeled data can be used to generate a test data set to test the trained model, and a classification model with better performance index values ​​can be obtained, ensuring that the text classification model obtained based on the continuously updated training data can adapt to the needs of ever-changing business scenarios; 3) A complete classification model automatic training and deployment mechanism is designed. Due to the continuously updated training data, it is only necessary to set the training method, training trigger method, and service deployment conditions. When the training trigger method is established, the model can be automatically trained according to the set training method, and the trained classification model can be tested. Then, when the service deployment conditions are established, the service is automatically deployed, thus forming an automatic cycle of text classification model training and continuous deployment, simplifying the operation of operation and maintenance personnel and improving operation and maintenance efficiency.

[0051] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0053] Figure 1 It is a flowchart of a method for incremental training and continuous deployment of a text classification model provided by an exemplary embodiment of the present invention;

[0054] Figure 2 It is a structural diagram of a system for incremental training and continuous deployment of a text classification model provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0055] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0056] It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0057] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0058] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0059] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0060] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0061] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0062] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0063] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0064] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0065] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0066] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0067] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0068] Exemplary Methods

[0069] Figure 1 is a flow chart of a method for incremental training and continuous deployment of a text classification model provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the method for incremental training and continuous deployment of the text classification model described in this embodiment includes the following steps:

[0070] Step 101: Generate feedback data according to the text data to be classified input by the user.

[0071] Optionally, generating feedback data according to the text data to be classified input by the user includes:

[0072] For the text data to be classified input by the user, the current text classification model deployed by the service is called to generate multiple classification results;

[0073] When the feedback data generation mode is passive feedback, one classification result is selected from the multiple classification results as the feedback data; when the feedback data generation mode is active feedback, the multiple classification results are discarded and a new classification result is generated as the feedback data.

[0074] In one embodiment, an interface for calling a text classification model service and a feedback interface are provided. When a user inputs text data to be classified, multiple classification results can be obtained by calling the interface of the text classification model service. When the feedback data generation mode is passive feedback, the user selects the most suitable result from multiple classification results as feedback data. When the feedback data generation mode is active feedback, the user inputs the correct classification result, and the input classification result is used as feedback data. By generating feedback data for the text data to be classified input by the user, a continuous source of training data is provided for model training.

[0075] In step 102, according to the selected model training method, the full training data set, the incremental training data set and the test data set are updated based on the text data to be classified, the feedback data and the imported annotation data.

[0076] Optionally, according to the selected model training method, updating the full training data set, the incremental training data set, and the test data set based on the text data to be classified, the feedback data, and the imported annotation data includes:

[0077] When the model training mode is incremental training, the text data to be classified, the feedback data and the first labeled data are added to the incremental training data set, wherein the imported labeled data includes the first labeled data and the second labeled data, and the initial value of the cardinality of the incremental training data set is 0;

[0078] When the model training mode is full training, the elements in the incremental training data set and the first labeled data are added to the full training data set according to the data addition trigger strategy, wherein the data addition trigger strategy includes timed trigger, quantitative trigger and manual trigger. The quantitative trigger means that data addition is performed after the cardinality of the incremental training data set reaches the set update trigger threshold. The initial value of the cardinality of the full training data set is 0.

[0079] When performing incremental training or full training, the second labeled data is added to the test data set, wherein the initial value of the cardinality of the test data set is 0.

[0080] In one embodiment, the text data to be classified and the corresponding feedback data input by the user are obtained by calling the feedback interface as the data source of the incremental training data set for incremental training. Furthermore, for the imported annotation data, it can also be used as one of the data sources of the incremental training data set during incremental training. When the model training method is full training, the first annotation data imported before the full training and all elements in the incremental training data set are used as the data source of the full training data set. Regardless of whether incremental training or full training is performed, as long as the classification model training is completed and the pending text classification model is generated, the pending text classification model needs to be tested, and the second annotation data imported before the test is directly used as the data source of the test data set. The data in the test data set is standard data that has been manually annotated, and the use of the test data set can accurately detect the performance of the model.

[0081] In step 103, based on the incremental training data set, the full training data set and the test data set, a plurality of first text classification models are generated according to pre-set model training rules.

[0082] Optionally, according to the selected model training method, determine the initial values ​​of the model parameters of the training text classification model;

[0083] According to the initial values ​​of the model parameters, a training data set corresponding to the model training method is used, and model training is performed according to the selected model training trigger method to generate multiple pending text classification models, wherein when the model training method is full training, the full training data set is used; when the model training method is incremental training, the incremental training data set is used; when the model training trigger method is manual triggering, the model training is manually triggered; when the model training trigger method is periodic triggering, the model training is triggered according to the set training period and training method; when the model training trigger method is indicator decline triggering, the model training is triggered when it is detected that the model performance indicator declines to the set training trigger threshold;

[0084] For each generated pending text classification model, a test data set is used to detect the model performance. When the model performance index value of the pending text classification model is greater than a preset performance threshold, the pending text classification model is determined to be the first text classification model. When the model performance index value of the pending text classification model is not greater than the preset performance threshold, the pending text classification model is discarded.

[0085] In one embodiment, two training methods are designed: full training and incremental training. In the full training method, the module uses all the data in the full training data set for training, and the initial value of the model parameter is a random number. In the incremental training method, the module uses the data in the incremental training data set for training, and the initial value of the model parameter is the parameter value of the previous model. The previous model is obtained by reading the model meta-information data, wherein the model meta-information data is the basic information recorded during model training, and the information includes training data version and training data snapshot, training hyperparameters, training start and end time, training log, test set indicators, model file storage path, model online record and other information. The model training parameters are saved in the configuration file, which records the hyperparameter settings and training data sets used for training the model, and specifies the use of full or incremental training. Model training can be performed after the training script reads the configuration file to set the relevant parameters.

[0086] In another embodiment, three triggering modes for automatic model training are designed: manual triggering, periodic triggering, and indicator decline triggering. Among them, manual triggering is to manually specify immediate model training according to demand. The periodic triggering mode is to pre-set the training period and training mode, for example, incremental training is performed once a day, and full training is performed once every 10 days. Indicator decline triggering means that when a decline in model indicators is detected, full training is immediately triggered.

[0087] In step 104, according to the preset model deployment trigger condition, a first text classification model that meets the model deployment trigger condition is selected from multiple first text classification models as a second text classification model for service deployment.

[0088] Optionally, selecting, according to a preset model deployment trigger condition, a first text classification model satisfying the model deployment trigger condition from a plurality of first text classification models as a second text classification model for service deployment includes:

[0089] When the model deployment trigger condition is to query the first text classification model with the highest model performance index value within a time period, the model performance index values ​​of the multiple first text classification models are sorted from large to small;

[0090] The first text classification model ranked first is selected as the second text classification model for service deployment.

[0091] Optionally, the method also includes detecting the availability of a second text classification model used for service deployment; when the second text classification model is available, switching the external access traffic of the service to the second text classification model, and stopping the current text classification model used for service deployment; when the second text classification model is unavailable, generating a warning message.

[0092] In one embodiment, taking Docker deployment as an example, first modify the deployment file, update the model file path, the file path points to the second text classification model, then start the Docker container, and run the second text classification model service. Finally, detect the availability of the second text classification model service. If available, switch the external access traffic to the second text classification model service and stop the old model service. If it is detected that the second text classification model service is unavailable, stop the deployment and send an alarm email to the administrator.

[0093] The method further includes calculating user satisfaction according to user behavior data for a second text classification model used for service deployment, and determining a performance level of the second text classification model according to a correspondence between a user satisfaction interval and a model performance level, wherein:

[0094] The formula for calculating user satisfaction S based on user behavior data is:

[0095] S = (number of active feedbacks + number of passive feedbacks) / number of times the second text classification model is called.

[0096] Exemplary Systems

[0097] Figure 2 Schematic diagram of the structure of a system for incremental training and continuous deployment of a text classification model provided by an exemplary embodiment of the present invention. Figure 2 As shown, the system for incremental training and continuous deployment of the text classification model described in this embodiment includes:

[0098] Feedback data module 201, used to generate feedback data according to the text data to be classified input by the user;

[0099] The data processing module 202 is used to update the full training data set, the incremental training data set and the test data set based on the text data to be classified, the feedback data and the imported annotation data according to the selected model training method;

[0100] A model training module 203 is used to generate a plurality of first text classification models based on the incremental training data set, the full training data set and the test data set according to a preset model training rule;

[0101] The model deployment module 204 is used to select a first text classification model that meets the model deployment trigger condition from multiple first text classification models as a second text classification model for service deployment according to a preset model deployment trigger condition.

[0102] Optionally, the model deployment module 204 also detects the availability of a second text classification model used for service deployment. When the second text classification model is available, the external access traffic of the service is switched to the second text classification model, and the current text classification model used for service deployment is stopped; when the second text classification model is unavailable, a warning message is generated.

[0103] Optionally, the feedback data module 201 generates feedback data according to the text data to be classified input by the user, including:

[0104] For the text data to be classified input by the user, the current text classification model deployed by the service is called to generate multiple classification results;

[0105] When the feedback data generation mode is passive feedback, one classification result is selected from the multiple classification results as the feedback data; when the feedback data generation mode is active feedback, the multiple classification results are discarded and a new classification result is generated as the feedback data.

[0106] Optionally, the data processing module 202 updates the full training data set, the incremental training data set, and the test data set based on the text data to be classified, the feedback data, and the imported annotation data according to the selected model training method, including:

[0107] When the model training mode is incremental training, the text data to be classified, the feedback data and the first labeled data are added to the incremental training data set, wherein the imported labeled data includes the first labeled data and the second labeled data, and the initial value of the cardinality of the incremental training data set is 0;

[0108] When the model training mode is full training, the elements in the incremental training data set and the first labeled data are added to the full training data set according to the data addition trigger strategy, wherein the data addition trigger strategy includes timed trigger, quantitative trigger and manual trigger. The quantitative trigger means that data addition is performed after the cardinality of the incremental training data set reaches the set update trigger threshold. The initial value of the cardinality of the full training data set is 0.

[0109] When performing incremental training or full training, the second labeled data is added to the test data set, wherein the initial value of the cardinality of the test data set is 0.

[0110] Optionally, the model training module 203 generates a plurality of first text classification models based on the incremental training data set, the full training data set and the test data set according to a preset model training rule, including:

[0111] Determine the initial values ​​of the model parameters of the training text classification model according to the selected model training method;

[0112] According to the initial values ​​of the model parameters, a training data set corresponding to the model training method is used, and model training is performed according to the selected model training trigger method to generate multiple pending text classification models, wherein when the model training method is full training, the full training data set is used; when the model training method is incremental training, the incremental training data set is used; when the model training trigger method is manual triggering, the model training is manually triggered; when the model training trigger method is periodic triggering, the model training is triggered according to the set training period and training method; when the model training trigger method is indicator decline triggering, the model training is triggered when it is detected that the model performance indicator declines to the set training trigger threshold;

[0113] For each generated pending text classification model, a test data set is used to detect the model performance. When the model performance index value of the pending text classification model is greater than a preset performance threshold, the pending text classification model is determined to be the first text classification model. When the model performance index value of the pending text classification model is not greater than the preset performance threshold, the pending text classification model is discarded.

[0114] Optionally, the service deployment module 204 selects a first text classification model that meets the model deployment trigger condition from multiple first text classification models as the second text classification model for service deployment according to a preset model deployment trigger condition, including:

[0115] When the model deployment trigger condition is to query the first text classification model with the highest model performance index value within a time period, the model performance index values ​​of the multiple first text classification models are sorted from large to small;

[0116] The first text classification model ranked first is selected as the second text classification model for service deployment.

[0117] Optionally, the system further comprises a satisfaction evaluation module, which calculates user satisfaction according to user behavior data for the second text classification model used for service deployment, and determines the performance level of the second text classification model according to the corresponding relationship between the user satisfaction interval and the model performance level, wherein:

[0118] The formula for calculating user satisfaction S based on user behavior data is:

[0119] S = (number of active feedbacks + number of passive feedbacks) / number of times the second text classification model is called.

[0120] from Figure 2It can be seen that the system for incremental training and continuous deployment of text classification models provided in this embodiment forms a complete loop structure in terms of training data sources, model training, model deployment, etc., and uses feedback data as one of the sources of annotation data, which not only reduces the cost of manual annotation, but also ensures the sustainability of annotation data, and ensures that the text classification model obtained based on the continuously updated training data can adapt to the needs of ever-changing business scenarios; and the designed complete classification model automatic training and deployment mechanism, due to the continuously updated training data, only needs to set the training method, training trigger method, and service deployment conditions, and when the training trigger method is established, the model training can be automatically carried out according to the set training method, and the trained classification model can be tested, and then when the service deployment conditions are established, the service deployment is automatically carried out, thereby forming an automatic cycle of text classification model training and continuous deployment, simplifying the operation of operation and maintenance personnel and improving operation and maintenance efficiency.

[0121] Exemplary computer program products and computer-readable storage media

[0122] In addition to the above-mentioned methods and apparatus, an embodiment of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the method for incremental training and continuous deployment of a text classification model according to various embodiments of the present disclosure described in the above-mentioned “Exemplary Method” section of this specification.

[0123] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0124] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, causes the processor to execute the steps of the method for incremental training and continuous deployment of a text classification model according to various embodiments of the present disclosure described in the above “Exemplary Method” section of this specification.

[0125] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0126] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, and are not limitations. The above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0127] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0128] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0129] The method and apparatus of the present disclosure may be implemented in many ways. For example, the method and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0130] It should also be noted that in the apparatus, equipment and method of the present disclosure, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present disclosure. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present disclosure. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the aspects shown here, but to the widest scope consistent with the principles and novel features disclosed herein.

[0131] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A method for incremental training and continuous deployment of a text classification model, characterized in that: The method comprises: Generate feedback data according to the text data to be classified input by the user; According to the selected model training method, the full training dataset, incremental training dataset, and test dataset are updated based on the text data to be classified, feedback data, and imported annotation data, including: When the model training mode is incremental training, the text data to be classified, the feedback data and the first labeled data are added to the incremental training data set, wherein the imported labeled data includes the first labeled data and the second labeled data, and the initial value of the cardinality of the incremental training data set is 0; When the model training mode is full training, the elements in the incremental training data set and the first labeled data are added to the full training data set according to the data addition trigger strategy, wherein the data addition trigger strategy includes timed trigger, quantitative trigger and manual trigger. The quantitative trigger means that data addition is performed after the cardinality of the incremental training data set reaches the set update trigger threshold. The initial value of the cardinality of the full training data set is 0. When performing incremental training or full training, the second labeled data is added to the test data set, wherein the initial value of the cardinality of the test data set is 0; Based on the incremental training data set, the full training data set and the test data set, and according to pre-set model training rules, multiple first text classification models are generated; According to the preset model deployment trigger condition, a first text classification model that meets the model deployment trigger condition is selected from multiple first text classification models as the second text classification model for service deployment.

2. The method according to claim 1, characterized in that: The method also includes detecting the availability of a second text classification model used for service deployment; when the second text classification model is available, switching the external access traffic of the service to the second text classification model and stopping the current text classification model used for service deployment; and when the second text classification model is unavailable, generating a warning message.

3. The method according to claim 1, characterized in that The feedback data generated based on the text data to be classified input by the user includes: For the text data to be classified input by the user, the current text classification model deployed by the service is called to generate multiple classification results; When the feedback data generation mode is passive feedback, one classification result is selected from the multiple classification results as the feedback data; when the feedback data generation mode is active feedback, the multiple classification results are discarded and a new classification result is generated as the feedback data.

4. The method according to claim 1, characterized in that: Based on the incremental training data set, the full training data set and the test data set, generating multiple first text classification models according to preset model training rules includes: Determine the initial values ​​of the model parameters of the training text classification model according to the selected model training method; According to the initial values ​​of the model parameters, a training data set corresponding to the model training method is used, and model training is performed according to the selected model training trigger method to generate multiple pending text classification models, wherein when the model training method is full training, the full training data set is used; when the model training method is incremental training, the incremental training data set is used; when the model training trigger method is manual triggering, the model training is manually triggered; when the model training trigger method is periodic triggering, the model training is triggered according to the set training period and training method; when the model training trigger method is indicator decline triggering, the model training is triggered when it is detected that the model performance indicator declines to the set training trigger threshold; For each generated pending text classification model, a test data set is used to detect the model performance. When the model performance index value of the pending text classification model is greater than a preset performance threshold, the pending text classification model is determined to be the first text classification model. When the model performance index value of the pending text classification model is not greater than the preset performance threshold, the pending text classification model is discarded.

5. The method according to claim 1, characterized in that According to a preset model deployment trigger condition, selecting a first text classification model that meets the model deployment trigger condition from a plurality of first text classification models as a second text classification model for service deployment includes: When the model deployment trigger condition is to query the first text classification model with the highest model performance index value within a time period, the model performance index values ​​of the multiple first text classification models are sorted from large to small; The first text classification model ranked first is selected as the second text classification model for service deployment.

6. The method according to claim 1, characterized in that The method further includes calculating user satisfaction according to user behavior data for a second text classification model used for service deployment, and determining a performance level of the second text classification model according to a correspondence between a user satisfaction interval and a model performance level, wherein: The formula for calculating user satisfaction S based on user behavior data is: S = (number of active feedback + number of passive feedback) / number of times the second text classification model is called.

7. A system for incremental training and continuous deployment of text classification models, characterized in that: The system comprises: A feedback data module, used to generate feedback data according to the text data to be classified input by the user; The data processing module is used to update the full training data set, incremental training data set and test data set based on the text data to be classified, feedback data and imported annotation data according to the selected model training method, including: When the model training mode is incremental training, the text data to be classified, the feedback data and the first labeled data are added to the incremental training data set, wherein the imported labeled data includes the first labeled data and the second labeled data, and the initial value of the cardinality of the incremental training data set is 0; When the model training mode is full training, the elements in the incremental training data set and the first labeled data are added to the full training data set according to the data addition trigger strategy, wherein the data addition trigger strategy includes timed trigger, quantitative trigger and manual trigger. The quantitative trigger means that data addition is performed after the cardinality of the incremental training data set reaches the set update trigger threshold. The initial value of the cardinality of the full training data set is 0. When performing incremental training or full training, the second labeled data is added to the test data set, wherein the initial value of the cardinality of the test data set is 0; A model training module, used to generate a plurality of first text classification models based on an incremental training data set, a full training data set and a test data set according to a preset model training rule; The model deployment module is used to select a first text classification model that meets the model deployment trigger condition from multiple first text classification models as a second text classification model for service deployment according to a preset model deployment trigger condition.

8. The system according to claim 7, characterized in that The model deployment module also detects the availability of a second text classification model used for service deployment. When the second text classification model is available, the external access traffic of the service is switched to the second text classification model, and the current text classification model used for service deployment is stopped; when the second text classification model is unavailable, a warning message is generated.

9. The system according to claim 7, characterized in that The feedback data module generates feedback data based on the text data to be classified input by the user, including: For the text data to be classified input by the user, the current text classification model deployed by the service is called to generate multiple classification results; When the feedback data generation mode is passive feedback, one classification result is selected from the multiple classification results as the feedback data; when the feedback data generation mode is active feedback, the multiple classification results are discarded and a new classification result is generated as the feedback data.

10. The system according to claim 7, characterized in that The model training module generates multiple first text classification models based on the incremental training data set, the full training data set and the test data set according to the preset model training rules, including: Determine the initial values ​​of the model parameters of the training text classification model according to the selected model training method; According to the initial values ​​of the model parameters, a training data set corresponding to the model training method is used, and model training is performed according to the selected model training trigger method to generate multiple pending text classification models, wherein when the model training method is full training, the full training data set is used; when the model training method is incremental training, the incremental training data set is used; when the model training trigger method is manual triggering, the model training is manually triggered; when the model training trigger method is periodic triggering, the model training is triggered according to the set training period and training method; when the model training trigger method is indicator decline triggering, the model training is triggered when it is detected that the model performance indicator declines to the set training trigger threshold; For each generated pending text classification model, a test data set is used to detect the model performance. When the model performance index value of the pending text classification model is greater than a preset performance threshold, the pending text classification model is determined to be the first text classification model. When the model performance index value of the pending text classification model is not greater than the preset performance threshold, the pending text classification model is discarded.

11. The system according to claim 7, characterized in that The service deployment module selects a first text classification model that meets the model deployment trigger condition from multiple first text classification models as a second text classification model for service deployment according to a preset model deployment trigger condition, including: When the model deployment trigger condition is to query the first text classification model with the highest model performance index value within a time period, the model performance index values ​​of the multiple first text classification models are sorted from large to small; The first text classification model ranked first is selected as the second text classification model for service deployment.

12. The system according to claim 7, characterized in that The system further includes a satisfaction evaluation module, which calculates user satisfaction according to user behavior data for the second text classification model used for service deployment, and determines the performance level of the second text classification model according to the corresponding relationship between the user satisfaction interval and the model performance level, wherein: The formula for calculating user satisfaction S based on user behavior data is: S = (number of active feedback + number of passive feedback) / number of times the second text classification model is called.

Images