Classification model intelligent progressive training system and method
Through the intelligent progressive training system and human-computer interaction module, the labeling rule model is optimized, and the problem of low labeling accuracy and efficiency of classification model data is solved, and more efficient model training is achieved.
Patent Information
- Application Number
- CN202411064166.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the data labeling accuracy of the classification model is poor and has low efficiency, which affects the model training effect and efficiency.
Provide an intelligent progressive training system for classification models, including data acquisition module, labeling module, model training module, reference data set determination module and human-computer interaction module. The labeling data is marked through preset labeling rules and filter out the non-compliant rules. The data set of the data set and the adjustment instructions input by the user are iteratively optimized the annotation rule model to improve the annotation accuracy and efficiency.
By quickly iterating the annotation rule model, the accuracy and efficiency of data annotation are improved, and the accuracy and efficiency of model training are improved.
Smart Images

Figure CN119026091A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data annotation, and in particular to a classification model intelligent progressive training system and method. Background Art
[0002] Natural language processing is an important branch of artificial intelligence, which involves the technology of computer understanding, parsing and generating human language. In many applications of natural language processing, such as text classification, entity recognition and other tasks, it is usually necessary to annotate a large amount of corpus. However, manual annotation of corpus is time-consuming and labor-intensive, especially in the scenario where there is only unlabeled data, the cost and time cost of manual annotation are huge.
[0003] To solve this problem, researchers have proposed many automated data labeling methods. Among them, a common method is to use a pre-trained language model for data pre-labeling. This method trains a language model based on a large amount of labeled corpus, and then uses this model to pre-label the unlabeled corpus. However, the pre-trained language model may lead to inaccurate pre-labeling results due to the bias of the training data or the limitations of the model. Another common method is to use a linguistics-based system approach for rule modeling. This method relies on experts' in-depth understanding and summary of language rules, and annotates the corpus by defining a set of rule systems. However, rule modeling methods often require a lot of expert knowledge and manual input, and when faced with complex and changing language phenomena, rule modeling methods may not cover all situations.
[0004] In general, the current model training process, especially the classification model training process, suffers from poor data labeling accuracy and low efficiency, which affects the effect and efficiency of model training. Summary of the invention
[0005] The purpose of the present invention is to provide a classification model intelligent progressive training system and method to alleviate the technical problems of poor data labeling accuracy and low efficiency in the prior art and to improve the accuracy and efficiency of data labeling.
[0006] In the first aspect, an embodiment of the present invention provides an intelligent progressive training system for a classification model, comprising: a data acquisition module, a labeling module, a model training module, a reference data set determination module, a human-computer interaction module and an iterative training module connected in sequence; the data acquisition module is used to acquire the data to be labeled, a preset initial deep learning model and a preset labeling rule model; the labeling module is used to label the target data in the data to be labeled based on the preset labeling rules and the labeling rule model to obtain the labeled data; the model training module is used to train the initial deep learning model through the labeled data until the preset training end requirements are met to obtain the trained deep learning model. deep learning model; the reference data set determination module is used to determine the reference data set based on the preset data screening rules, according to the data to be labeled, the labeled data and the preset labeling rules; the data screening rules are constructed based on the labeling rule model and the trained deep learning model; the reference data set is used to indicate the data set that does not conform to the preset labeling rules in the data to be labeled and the labeled data; the human-computer interaction module is used to output the reference data set and obtain the parameter adjustment instructions of the labeling rule model input by the user; the iterative training module is used to adjust the parameters of the labeling rule model according to the parameter adjustment instructions to obtain the classification model.
[0007] In a preferred embodiment, the above-mentioned model training module is also used to obtain updated preset labeling rules after the step of adjusting the parameters of the above-mentioned labeling rule model according to the above-mentioned parameter adjustment instructions to obtain the classification model; based on the above-mentioned updated preset labeling rules, the updated target data in the above-mentioned data to be labeled is labeled through the above-mentioned labeling rule model to obtain updated labeling data; the above-mentioned trained deep learning model is retrained through the above-mentioned updated labeling data until the above-mentioned training end requirements are met to obtain an updated deep learning model.
[0008] In a preferred embodiment, the above-mentioned labeling rule model is designed based on a class network structure.
[0009] In a preferred embodiment, the above-mentioned network-like structure includes: an extraction layer, a judgment layer and an aggregation layer; the above-mentioned extraction layer is used to extract the above-mentioned data to be labeled according to a preset number of extractions and the extraction content corresponding to the above-mentioned extraction number, so as to obtain the extracted data; the above-mentioned judgment layer is used to screen the matching data that matches the preset semantic judgment rules from the above-mentioned extracted data; the above-mentioned aggregation layer is used to label the above-mentioned matching data based on the preset label assignment rules, so as to obtain the above-mentioned labeled data.
[0010] In a preferred embodiment, the reference data set determination module is also used to: based on a preset first data screening rule, screen out a first data set that meets the above-mentioned marking rules but has not been marked from the above-mentioned data to be marked; based on a preset second data screening rule, screen out a second data set whose processing results of the above-mentioned trained deep learning model are inconsistent with the processing results of the above-mentioned marking rule model from the marked data; based on a preset third data screening rule, screen out a third data set whose prediction probability is less than a preset threshold from the processing results of the above-mentioned trained deep learning model from the marked data; based on a preset fourth data screening rule, screen out a third data set that has been marked multiple times based on the above-mentioned preset marking rules from the marked data. A fourth data set; based on a preset fifth data screening rule, a fifth data set that meets the error rate requirement of the preset marking rule is screened out from the above-mentioned data to be labeled and the labeled data; based on a preset sixth data screening rule, a sixth data set that meets the preset proportion hit number requirement of the preset marking rule is screened out from the above-mentioned data to be labeled and the labeled data; based on a preset seventh data screening rule, a seventh data set whose label association number reaches a preset association threshold is screened out from the labeled data; the above-mentioned reference data set is determined based on the above-mentioned first data set, the above-mentioned second data set, the above-mentioned third data set, the above-mentioned fourth data set, the above-mentioned fifth data set, the above-mentioned sixth data set and the above-mentioned seventh data set.
[0011] In a preferred embodiment, the reference data set determination module is also used to randomly extract a preset proportion of data from the reference data set to obtain a data sample set; the human-computer interaction module is also used to output the data sample set and obtain parameter adjustment instructions for the labeling rule model input by the user.
[0012] In a preferred embodiment, the reference data set determination module is also used to extract data features of the data sample set based on preset feature extraction rules, and to count statistical information of the data sample set based on preset data statistical rules; the human-computer interaction module is also used to output the data features and the statistical information, and obtain parameter adjustment instructions of the labeling rule model input by the user.
[0013] In a preferred embodiment, the human-computer interaction module is also used to send the data features and the statistical information to a designated terminal for display.
[0014] In the second aspect, an embodiment of the present invention provides an intelligent progressive training method for a classification model, comprising: obtaining data to be labeled, a preset initial deep learning model, and a preset labeling rule model; based on the preset labeling rules, labeling the target data in the data to be labeled through the above-mentioned labeling rule model to obtain labeled data; training the above-mentioned initial deep learning model through the above-mentioned labeled data until the preset training end requirements are met to obtain a trained deep learning model; based on the preset data screening rules, determining a reference data set according to the above-mentioned data to be labeled, the above-mentioned labeled data, and the above-mentioned preset labeling rules; the above-mentioned data screening rules are constructed based on the above-mentioned labeling rule model and the trained deep learning model; the above-mentioned reference data set is used to indicate the data set in the above-mentioned data to be labeled and the above-mentioned labeled data that does not conform to the above-mentioned preset labeling rules; outputting the above-mentioned reference data set, and obtaining the parameter adjustment instructions of the above-mentioned labeling rule model input by the user; adjusting the parameters of the above-mentioned labeling rule model according to the above-mentioned parameter adjustment instructions to obtain a classification model.
[0015] In a preferred embodiment, after adjusting the parameters of the above-mentioned labeling rule model according to the above-mentioned parameter adjustment instruction to obtain the classification model, the above-mentioned method also includes: obtaining updated preset labeling rules; based on the above-mentioned updated preset labeling rules, labeling the updated target data in the above-mentioned data to be labeled through the above-mentioned labeling rule model to obtain updated labeling data; retraining the above-mentioned trained deep learning model through the above-mentioned updated labeling data until the above-mentioned training end requirements are met to obtain an updated deep learning model.
[0016] The embodiments of the present invention have the following beneficial technical effects:
[0017] An intelligent progressive training system and method for a classification model provided by an embodiment of the present invention comprises: a data acquisition module, a labeling module, a model training module, a reference data set determination module, a human-computer interaction module and an iterative training module connected in sequence; the data acquisition module is used to acquire data to be labeled, a preset initial deep learning model and a preset labeling rule model; the labeling module is used to label target data in the data to be labeled based on preset labeling rules and through the labeling rule model to obtain labeled data; the model training module is used to train the initial deep learning model through the labeled data until the preset training end requirement is met to obtain a trained deep Learning model; the reference data set determination module is used to determine the reference data set based on the preset data screening rules, according to the data to be labeled, the labeled data and the preset labeling rules; the data screening rules are constructed based on the labeling rule model and the trained deep learning model; the reference data set is used to indicate the data set that does not conform to the preset labeling rules in the data to be labeled and the labeled data; the human-computer interaction module is used to output the reference data set and obtain the parameter adjustment instructions of the labeling rule model input by the user; the iterative training module is used to adjust the parameters of the labeling rule model according to the parameter adjustment instructions to obtain the classification model. The system feeds back the reference data set to the operator so that the operator can adjust the parameters of the labeling rule model, thereby realizing the rapid iteration of the labeling rule model, improving the accuracy and efficiency of data labeling, thereby improving the efficiency of model training, and improving the accuracy of the trained model in solving problems.
[0018] Other features and advantages disclosed in this embodiment will be described in the subsequent description, or some features and advantages can be inferred or determined without doubt from the description, or can be learned by implementing the above-mentioned technology of the present disclosure.
[0019] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0021] Figure 1 A schematic diagram of the structure of a classification model intelligent progressive training system provided by an embodiment of the present invention;
[0022] Figure 2 A schematic diagram of the structure of a labeling rule model provided by an embodiment of the present invention;
[0023] Figure 3 A schematic diagram of a flow chart of an intelligent progressive training method for a classification model provided in an embodiment of the present invention.
[0024] Icons: 11-data acquisition module; 12-labeling module; 13-model training module; 14-reference data set determination module; 15-human-computer interaction module; 16-iterative training module; 21-extraction layer; 22-judgment layer; 23-aggregation layer. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0026] At present, a common method in the prior art for realizing data labeling automation is to use a pre-trained language model for data pre-labeling. This method trains a language model based on a large amount of labeled corpus, and then uses this model to pre-label the unlabeled corpus. However, the pre-trained language model may cause the pre-labeling results to be inaccurate due to the deviation of the training data or the limitations of the model. Another common method is to use a linguistics-based system method for rule modeling. This method relies on the expert's in-depth understanding and summary of language rules, and annotates the corpus by defining a set of rule systems. However, the rule modeling method often requires a lot of expert knowledge and manual input, and when faced with complex and changing language phenomena, the rule modeling method may not cover all situations. Overall, the data labeling involved in the current model training process has the current status of poor data labeling accuracy and low efficiency. This has an adverse effect on the model obtained by training the labeled data obtained by these methods, such as affecting the efficiency of model training and the accuracy of the model in solving problems.
[0027] Based on this, an embodiment of the present invention provides a classification model intelligent progressive training system and method. The system can improve the accuracy and efficiency of data annotation, improve the efficiency of model training, and improve the accuracy of problem solving by the trained model. To facilitate understanding of the embodiment of the present invention, a classification model intelligent progressive training system disclosed in the embodiment of the present invention is first introduced in detail.
[0028] Example 1
[0029] In this embodiment, Figure 1 A schematic diagram of the structure of a classification model intelligent progressive training system provided in an embodiment of the present invention.
[0030] Depend on Figure 1 As can be seen, the system includes: a data acquisition module 11, a labeling module 12, a model training module 13, a reference data set determination module 14, a human-computer interaction module 15 and an iterative training module 16 which are connected in sequence.
[0031] Among them, the above-mentioned data acquisition module 11 is used to obtain the data to be labeled, the preset initial deep learning model and the preset labeling rule model; the above-mentioned labeling module 12 is used to label the target data in the above-mentioned data to be labeled based on the preset labeling rules and the above-mentioned labeling rule model to obtain the labeled data; the above-mentioned model training module 13 is used to train the above-mentioned initial deep learning model through the above-mentioned labeled data until the preset training end requirements are met to obtain the trained deep learning model; the above-mentioned labeling module 12 is also used to label the target data in the data to be labeled through the trained deep learning model to obtain the labeled data; the above-mentioned reference data set determination module 14, used to determine the reference data set based on preset data screening rules, according to the above-mentioned data to be labeled, the above-mentioned labeled data and the above-mentioned preset labeling rules; the above-mentioned data screening rules are constructed based on the above-mentioned labeling rule model and the above-mentioned trained deep learning model; the above-mentioned reference data set determination module is used to indicate the data set that does not conform to the above-mentioned preset labeling rules in the above-mentioned data to be labeled and the above-mentioned labeled data; the above-mentioned human-computer interaction module 15 is used to output the above-mentioned reference data set and obtain the parameter adjustment instructions of the above-mentioned labeling rule model input by the user; the iterative training module 16 is used to adjust the parameters of the above-mentioned labeling rule model according to the above-mentioned parameter adjustment instructions to obtain a classification model.
[0032] It should be noted that different annotation rule models involve different parameters, so the specific parameters are determined by the specific annotation rule model.
[0033] By comparing the results of the annotation rule model and the trained deep learning model, various data problems can be discovered and processed, thereby helping modeling engineers optimize rule modeling (for example, how to adjust the parameters of the annotation rule model).
[0034] In one embodiment, the model training module 13 is also used to obtain updated preset labeling rules after the step of adjusting the parameters of the labeling rule model according to the parameter adjustment instruction to obtain the classification model; based on the updated preset labeling rules, the updated target data in the data to be labeled is labeled by the labeling rule model to obtain updated labeling data; the trained deep learning model is retrained by the updated labeling data until the training end requirement is met to obtain an updated deep learning model. The data to be labeled is labeled by the labeling rule model with adjusted parameters and the updated deep learning model, and then the reference data set is determined by comparing the results of the two labels, so as to further adjust the parameters of the labeling rule model, thereby improving the accuracy and efficiency of data labeling.
[0035] Furthermore, the above-mentioned labeling rule model is designed based on a class network structure.
[0036] For ease of understanding, Figure 2 A schematic diagram of the structure of a labeling rule model provided by an embodiment of the present invention.
[0037] Depend on Figure 2 As can be seen, the above-mentioned network-like structure includes: an extraction layer 21, a judgment layer 22 and an aggregation layer 23; the above-mentioned extraction layer 21 is used to extract the above-mentioned data to be labeled according to a preset number of extractions and the extraction content corresponding to the above-mentioned extraction times, so as to obtain the extracted data; the above-mentioned judgment layer 22 is used to screen the matching data that matches the preset semantic judgment rules from the above-mentioned extracted data; the above-mentioned aggregation layer 23 is used to label the above-mentioned matching data based on the preset label assignment rules, so as to obtain the above-mentioned labeled data.
[0038] Furthermore, the reference data set determination module 14 is further configured to:
[0039] The reference data set determination module is also used to: based on a preset first data screening rule, screen out a first data set that meets the above-mentioned marking rule but is not marked from the above-mentioned data to be marked; based on a preset second data screening rule, screen out a second data set whose processing result of the above-mentioned trained deep learning model is inconsistent with the processing result of the above-mentioned marking rule model from the marked data; based on a preset third data screening rule, screen out a third data set whose prediction probability is less than a preset threshold from the processing result of the above-mentioned trained deep learning model from the marked data; based on a preset fourth data screening rule, screen out a fourth data set that has been marked multiple times based on the above-mentioned preset marking rule from the marked data ; Based on the preset fifth data screening rule, a fifth data set that meets the error rate requirement of the preset marking rule is screened out from the above-mentioned data to be labeled and the labeled data; Based on the preset sixth data screening rule, a sixth data set that meets the preset proportion hit number requirement of the preset marking rule is screened out from the above-mentioned data to be labeled and the labeled data; Based on the preset seventh data screening rule, a seventh data set whose label association number reaches a preset association threshold is screened out from the labeled data; The above-mentioned reference data set is determined based on the above-mentioned first data set, the above-mentioned second data set, the above-mentioned third data set, the above-mentioned fourth data set, the above-mentioned fifth data set, the above-mentioned sixth data set and the above-mentioned seventh data set.
[0040] Here, the above-mentioned labeled data is obtained by labeling the same batch of data to be labeled based on the above-mentioned classification model and the trained deep learning model.
[0041] In actual operation, the above-mentioned first data set is generally named as rule miss set; the above-mentioned rule miss set includes data in the data to be labeled that meets the above-mentioned labeling rules but has not been labeled; the above-mentioned second data set is generally named as conflict set; the above-mentioned conflict set includes data in the above-mentioned labeled data whose processing results of the above-mentioned trained deep learning model are inconsistent with the processing results of the above-mentioned labeling rule model; the above-mentioned third data set is generally named as deep confusion set; the above-mentioned deep confusion set includes data in the processing results of the trained deep learning model whose prediction probability is less than a preset threshold; the above-mentioned fourth data set is generally named as rule confusion set; the above-mentioned rule confusion set includes data that has been labeled multiple times based on the above-mentioned preset labeling rules; the above-mentioned fifth data set is generally named as error rate sampling set; the above-mentioned error rate sampling set includes the above-mentioned data to be labeled and the error rate requirement data in the above-mentioned labeled data that meets the above-mentioned preset labeling rules; the above-mentioned sixth data set is generally named as clustered sampling set; the above-mentioned clustered sampling set includes the above-mentioned data to be labeled and the data in the above-mentioned labeled data that meet the preset proportion hit number requirement of the above-mentioned preset labeling rules; the above-mentioned seventh data set is generally named as label-based sampling set; the above-mentioned label-based sampling set includes data in the above-mentioned labeled data whose label association number reaches the preset association threshold.
[0042] In this embodiment, the rule miss set is expressed by the following formula:
[0043] [U=DR]
[0044] Among them, U is the rule miss set, D is the above data to be labeled, and R is the data labeled by the above labeling rule model.
[0045] Here, since the labeling rule model has high precision and low recall, there are situations where the defined rules do not cover them. Feeding this part of data back to the modeler will greatly help the modeler to complete the rules of the modeling model.
[0046] In this embodiment, the above conflict set is expressed by the following formula:
[0047] [C={d∈D[R(d)≠L(d)}]
[0048] Among them, C is the conflict set, R(d) is the result of the labeling rule model marking the element d in the labeled data, and L(d) is the result of the deep learning model marking the element d in the dataset.
[0049] Here, the data with different analysis results from the labeling rule model and the deep learning model are most likely on the edge of semantic concepts or have unclear definitions. This part of data can help modelers better revise the model and improve the accuracy of the modeling rules.
[0050] In this embodiment, the above depth confusion set is expressed by the following formula:
[0051] [DC={x∈D|P(x) <t}]
[0052] Among them, DC is the deep confusion set, D is the above-mentioned data to be labeled, P(x) is the predicted probability of the deep learning model for the sub-data x in the above-mentioned data to be labeled, and t is the preset threshold.
[0053] Here, the logit (logistic regression value) output by deep learning is a floating point value between 0 and 1. Generally speaking, for multi-classification, the label with the largest logit corresponding to the category is taken as the final label. For example, the above deep confusion set is a data set with a logit corresponding to the final label less than 0.9. This part of data represents that the deep learning model believes that it should belong to this category. This part of data helps modelers better understand the data cognition of the deep learning model and better provide a revised rule model to guide the deep learning model to develop in the expected way.
[0054] In this embodiment, the above rule confusion set is expressed by the following formula:
[0055] [RC={x∈D[N(x))>1}]
[0056] Among them, RC is the rule confusion set, D is the above data to be labeled, and N(x) is the number of different ontology categories hit by the sub-data x in the above data to be labeled.
[0057] Here, there will be many ontologies in the above annotation rule model, and each classification category corresponds to multiple ontologies. The data in the rule confusion set are all data that are assigned to multiple categories by the rule model. For example, for the target text, the rule model may consider it to be a "military" topic, and also a "financial" topic. However, the preset scenario is a classification scenario, and only one most appropriate type needs to be identified for the above target text. Therefore, determining the rule confusion set helps modelers to more standardize the boundary definition of the modeling ontology.
[0058] In this embodiment, the above error rate sampling set is expressed by the following formula:
[0059] [ERS r =Sample(D,E(r))]
[0060] Among them, Sample(D,E(r)) means sampling from the above-mentioned data to be labeled D according to the error rate E(r).
[0061] Furthermore, the error rate sampling set is sampled synchronously on the training set based on the error rates of the rule ontology under different categories in the test set. This part of data helps modelers understand the performance of rule modeling under different labels and have the most intuitive understanding of the ability distribution of rule models under different categories.
[0062] Furthermore, each rule ontology r in the annotation rule model can be expressed as the above formula.
[0063] In this embodiment, the above-mentioned aggregate sampling set is expressed by the following formula:
[0064] {ASS r =d∈D|dis sampled log(1+Hit(r))times for its associatedr∈R}
[0065] Among them, R is a set of rule ontology, each rule ontology r∈R has a hit number Hit(r), and each data d∈D in the data to be labeled D is associated with a rule ontology r.
[0066] Furthermore, the more data a rule ontology is associated with, the more biased the sampling will be towards the data corresponding to the rule ontology. This data set will intuitively reflect which modeling rules have a wider impact, and when modelers modify these rules, they will appropriately reduce the scope of modification to prevent excessive impact.
[0067] Furthermore, each rule r in the labeling rule model can be expressed as the above formula.
[0068] In this embodiment, the label-based sampling set is expressed by the following formula:
[0069] {LSS}_y=d in D|d is sampled Count(y)times for its associated y in Y}
[0070] Here, Count(y) represents the number of data associated with label in the above-mentioned data to be labeled D. In this way, for each label, sampling is performed according to its number in the data set. The data corresponding to labels with a large number will be sampled multiple times, and the data corresponding to labels with a small number will be sampled less.
[0071] Here, the label-based sampling set can reflect which labels have more data and which labels have less data, providing a reference for subsequent data analysis and model optimization.
[0072] Furthermore, the reference data set determination module 14 is also used to randomly extract a preset proportion of data from the reference data set to obtain a data sample set; the human-computer interaction module is also used to output the data sample set and obtain parameter adjustment instructions of the annotation rule model input by the user.
[0073] Furthermore, the reference data set determination module 14 is also used to extract data features of the data sample set based on preset feature extraction rules, and to count statistical information of the data sample set based on preset data statistical rules; the human-computer interaction module 15 is also used to output the data features and the statistical information, and obtain parameter adjustment instructions of the labeling rule model input by the user.
[0074] Furthermore, the human-computer interaction module 15 is also used to send the data features and the statistical information to a designated terminal for display.
[0075] Here, the human-computer interaction module 15 includes a display interface to display the data features and the statistical information through the display interface.
[0076] Furthermore, the above users can annotate the comparative data, add annotations, or adjust the modeling rules.
[0077] Here, the user can view the data of the above-mentioned rule miss set through the display interface, and modify the modeling rules of the labeling rule model according to the data characteristics of the rule miss set to enhance the recall rate of the modeling rules; the user can view the data of the above-mentioned conflict set through the display interface, and judge whether the rule recall is wrong or the deep learning model is wrong based on the label definition of the labeled data, and correct the rule recall error; the user can view the data of the above-mentioned deep confusion set through the display interface to see what characteristics the low-confidence data has. This type of data is essentially closer to the middle line of classification, that is, it is possible to have features of label A and label B at the same time. Optimizing the model based on these data can enhance the model's discrimination; the user can view the above-mentioned error rate sampling set, clustered sampling set and label-based sampling set through the display interface, so as to intuitively judge the effect of the model in actual use from a variety of different sampling sets, for example: see how many of the 100 data in the error rate sampling set, clustered sampling set and label-based sampling set are misclassified.
[0078] In actual operation, the parameters of the above-mentioned labeling rule model are adjusted through human-machine collaboration to obtain a classification model, which is a method that combines human expertise with machine intelligence, aiming to solve complex problems by effectively using machine learning and human expertise. Compared with manual labeling and manual rule modeling, it can effectively save labor costs and improve efficiency.
[0079] Furthermore, the present invention also provides the following experimental data:
[0080] Purpose of the experiment: Because the data is not fully labeled, the above labeling rule model is bound to have mislabeling, so it is necessary to know the impact of a certain amount of erroneous data sets on the final recognition effect of the model.
[0081] Experimental design: Four Chinese open source text classification datasets were pre-screened. Each open source text classification dataset has been precisely labeled, so the recognition accuracy of the labeling rule model can be evaluated based on the above open source text classification datasets.
[0082] Furthermore, a certain proportion of the training set data labels of these data sets are randomly shuffled, and the same deep learning model training method and hyperparameters are used for model training. The following are the experimental data sets and data experimental results.
[0083] The experimental data set is represented by the following Table 1:
[0084] Table 1
[0085] name Training set size Validation set size Number of categories Toutiao 382678 20000 15 THUCNews 378808 19989 14 FudanNews 301811 16762 12 IFlytek 6497 1137 14
[0086] The data experimental results are shown in Table 2 below:
[0087] Table 2
[0088] Dataset Toutiao THUCNews FudanNews IFlytek No confusion 92.94 87.99 89.90 79.67 5% confusion 93.07 87.96 89.97 79.16 10% confusion 91.68 88.14 89.57 79.67 Obfuscation 12% 90.43 88.02 89.06 79.87 Obfuscation 15% 90.45 87.59 89.27 79.81 Confusion 17% 90.68 87.80 88.97 79.45 20% confusion 89.37 87.48 88.90 79.08
[0089] Experimental conclusion: Confusion below 20% will not have much impact on model training in classification scenarios. As a rule-based modeling, multi-level modeling is characterized by extremely high precision and low recall. Therefore, the conclusion drawn from the above experiment is that the error rate caused by the annotation of multi-level modeling will not have much impact on the model training itself.
[0090] Furthermore, the invention also provides the following experiment for evaluating the effect of human-machine collaborative algorithm:
[0091] Purpose of the experiment: The reference data set needs to be returned because this data will have a higher reference value for optimization.
[0092] Experimental design: Use business data to conduct experiments, and count the error rates of data filtered out by preset recommendation strategies and the error rates of randomly filtered data.
[0093] The reference data set and data set error rate corresponding to the above business data are shown in Table 3 below:
[0094] Table 3
[0095]
[0096]
[0097] Experimental conclusion: From Table 3 above, we can see that the error rates of the seven selected sampling sets are all greater than the official error rates. It can be seen that the reference data set selected by this application is more valuable for modelers. Modelers can determine the parameter adjustment strategy for model training based on the reference data set, thereby achieving better training results.
[0098] Furthermore, the system continuously acquires and updates preset labeling rules, and based on the updated preset labeling rules, labels the updated target data in the above-mentioned data to be labeled through the above-mentioned labeling rule model to obtain updated labeling data, and then retrains the above-mentioned trained deep learning model through the above-mentioned updated labeling data until the above-mentioned training end requirements are met, and obtains an updated classification model, thereby realizing continuous iteration of the classification model.
[0099] An intelligent progressive training system and method for a classification model provided by an embodiment of the present invention comprises: a data acquisition module, a labeling module, a model training module, a reference data set determination module, a human-computer interaction module and an iterative training module connected in sequence; the data acquisition module is used to acquire data to be labeled, a preset initial deep learning model and a preset labeling rule model; the labeling module is used to label target data in the data to be labeled based on preset labeling rules and through the labeling rule model to obtain labeled data; the model training module is used to train the initial deep learning model through the labeled data until the preset training end requirement is met to obtain a trained deep learning model; the labeling module is also used to, through the trained deep The learning model labels the target data in the data to be labeled to obtain the labeled data; the reference data set determination module is used to determine the reference data set based on the preset data screening rules, according to the data to be labeled, the labeled data and the preset labeling rules; the data screening rules are constructed based on the labeling rule model and the trained deep learning model; the reference data set is used to indicate the data to be labeled and the data set that does not conform to the preset labeling rules in the labeled data; the human-computer interaction module is used to output the reference data set and obtain the parameter adjustment instructions of the labeling rule model input by the user; the iterative training module is used to adjust the parameters of the labeling rule model according to the parameter adjustment instructions to obtain the classification model. The system feeds back the reference data set to the operator so that the operator can adjust the parameters of the labeling rule model, thereby realizing the rapid iteration of the labeling rule model and improving the accuracy and efficiency of data labeling.
[0100] Example 2
[0101] Based on Example 1, Figure 3 A schematic diagram of a flow chart of an intelligent progressive training method for a classification model provided in an embodiment of the present invention.
[0102] Depend on Figure 3 As can be seen, the method includes:
[0103] Step S301: Obtain the data to be labeled, the preset initial deep learning model, and the preset labeling rule model.
[0104] Step S302: Based on preset labeling rules, target data in the data to be labeled is labeled using the labeling rule model to obtain labeled data.
[0105] Step S303: Train the initial deep learning model using the labeled data until the preset training end requirements are met to obtain a trained deep learning model.
[0106] Step S304: label the target data in the above-mentioned data to be labeled using the trained deep learning model to obtain labeled data.
[0107] Step S305: Based on preset data screening rules, determine the reference data set according to the above-mentioned data to be labeled, the above-mentioned labeled data and the above-mentioned preset labeling rules; the above-mentioned data screening rules are constructed based on the above-mentioned labeling rule model and the trained deep learning model; the above-mentioned reference data set is used to indicate the data set in the above-mentioned data to be labeled and the above-mentioned labeled data that does not comply with the above-mentioned preset labeling rules.
[0108] Step S306: output the reference data set and obtain the parameter adjustment instruction of the annotation rule model input by the user.
[0109] Step S307: adjusting the parameters of the above-mentioned labeling rule model according to the above-mentioned parameter adjustment instruction to obtain a classification model.
[0110] Furthermore, after the step of adjusting the parameters of the above-mentioned labeling rule model according to the above-mentioned parameter adjustment instruction to obtain the classification model, the above-mentioned method also includes: obtaining updated preset labeling rules; based on the above-mentioned updated preset labeling rules, labeling the updated target data in the above-mentioned data to be labeled through the above-mentioned labeling rule model to obtain updated labeling data; retraining the above-mentioned trained deep learning model through the above-mentioned updated labeling data until the above-mentioned training end requirements are met to obtain an updated deep learning model. The data to be labeled is labeled by the labeling rule model with adjusted parameters and the updated deep learning model, and then the reference data set is determined by comparing the results of the two labels to further adjust the parameters of the labeling rule model to obtain a classification model. The process of data labeling is essentially a classification process, and the labels of data labels can correspond to the categories of data.
[0111] Among them, step S305 includes: based on a preset first data screening rule, screening out a first data set that meets the above-mentioned marking rule but has not been marked from the above-mentioned data to be marked; based on a preset second data screening rule, screening out a second data set whose processing result of the above-mentioned trained deep learning model is inconsistent with the processing result of the above-mentioned marking rule model from the above-mentioned marked data; based on a preset third data screening rule, screening out a third data set whose prediction probability is less than a preset threshold from the processing result of the above-mentioned trained deep learning model from the above-mentioned marked data; based on a preset fourth data screening rule, screening out a fourth data set that has been marked multiple times based on the above-mentioned preset marking rule from the above-mentioned marked data; Based on the preset fifth data screening rule, a fifth data set that meets the error rate requirement of the preset marking rule is screened out from the above-mentioned data to be labeled and the above-mentioned labeled data; based on the preset sixth data screening rule, a sixth data set that meets the preset proportion hit number requirement of the above-mentioned preset marking rule is screened out from the above-mentioned data to be labeled and the above-mentioned labeled data; based on the preset seventh data screening rule, a seventh data set whose label association number reaches a preset association threshold is screened out from the above-mentioned labeled data; the above-mentioned reference data set is determined based on the above-mentioned first data set, the above-mentioned second data set, the above-mentioned third data set, the above-mentioned fourth data set, the above-mentioned fifth data set, the above-mentioned sixth data set and the above-mentioned seventh data set.
[0112] Furthermore, after step S305, the method further includes: randomly extracting a preset proportion of data from the reference data set to obtain a data sample set; outputting the data sample set, and obtaining a parameter adjustment instruction of the annotation rule model input by the user.
[0113] Furthermore, after step S305, the method also includes: extracting data features of the above-mentioned data sample set based on preset feature extraction rules, and counting statistical information of the above-mentioned data sample set based on preset data statistical rules; outputting the above-mentioned data features and the above-mentioned statistical information, and obtaining parameter adjustment instructions of the above-mentioned labeling rule model input by the user.
[0114] Furthermore, after step S305, the method further includes: sending the above data features and the above statistical information to a designated terminal for display.
[0115] Furthermore, the user can adjust the parameters of the labeling rule model according to the data characteristics and statistical information displayed by the terminal.
[0116] An embodiment of the present invention provides an intelligent progressive training method for a classification model, comprising: obtaining data to be labeled, a preset initial deep learning model and a preset labeling rule model; based on the preset labeling rules, labeling the target data in the data to be labeled through the labeling rule model to obtain labeled data; training the initial deep learning model through the labeled data until the preset training end requirements are met to obtain a trained deep learning model; labeling the target data in the data to be labeled through the trained deep learning model to obtain labeled data; based on the preset data screening rules, determining a reference data set according to the data to be labeled, the labeled data and the preset labeling rules; the data screening rules are constructed based on the labeling rule model and the trained deep learning model; the reference data set is used to indicate the data set that does not conform to the preset labeling rules in the data to be labeled and the labeled data; outputting the reference data set, and obtaining the parameter adjustment instruction of the labeling rule model input by the user; adjusting the parameters of the labeling rule model according to the parameter adjustment instruction to obtain a classification model. This method feeds back the reference data set to the operator so that the operator can adjust the parameters of the labeling rule model, thereby achieving rapid iteration of the labeling rule model and improving the accuracy and efficiency of data labeling.
Claims
1. A classification model intelligent progressive training system, characterized in that: include: A data acquisition module, a labeling module, a model training module, a reference data set determination module, a human-computer interaction module, and an iterative training module connected in sequence; The data acquisition module is used to acquire the data to be labeled, the preset initial deep learning model and the preset labeling rule model; The labeling module is used to label the target data in the data to be labeled based on the preset labeling rules and through the labeling rule model to obtain labeled data; The model training module is used to train the initial deep learning model through the labeled data until a preset training end requirement is met to obtain a trained deep learning model; The reference data set determination module is used to determine the reference data set based on the preset data screening rules, according to the data to be labeled, the labeled data and the preset labeling rules; the data screening rules are constructed based on the labeling rule model and the trained deep learning model; the reference data set is used to indicate the data set that does not conform to the preset labeling rules in the data to be labeled and the labeled data; The human-computer interaction module is used to output the reference data set and obtain the parameter adjustment instruction of the annotation rule model input by the user; The iterative training module is used to adjust the parameters of the labeling rule model according to the parameter adjustment instruction to obtain a classification model.
2. The classification model intelligent progressive training system according to claim 1, characterized in that: The model training module is also used to obtain updated preset labeling rules after the step of adjusting the parameters of the labeling rule model according to the parameter adjustment instruction to obtain the classification model; based on the updated preset labeling rules, the updated target data in the data to be labeled is labeled through the labeling rule model to obtain updated labeled data; The trained deep learning model is retrained by using the updated annotation data until the training end requirement is met to obtain an updated deep learning model.
3. The classification model intelligent progressive training system according to claim 1, characterized in that: The labeling rule model is designed based on a class network structure.
4. The classification model intelligent progressive training system according to claim 3, characterized in that: The network-like structure includes: an extraction layer, a judgment layer and an aggregation layer; The extraction layer is used to extract the data to be labeled according to a preset number of extractions and the extraction content corresponding to the number of extractions to obtain the extracted data; The judgment layer is used to screen the matching data that matches the preset semantic judgment rules from the extracted data; The collection layer is used to label the matching data based on a preset label assignment rule to obtain the labeled data.
5. The classification model intelligent progressive training system according to claim 1, characterized in that: The reference data set determination module is further used to: based on a preset first data screening rule, screen out a first data set that meets the marking rule but is not marked from the data to be marked; based on a preset second data screening rule, screen out a second data set whose processing result of the trained deep learning model is inconsistent with the processing result of the marking rule model from the marked data; based on a preset third data screening rule, screen out a third data set whose prediction probability is less than a preset threshold from the processing result of the trained deep learning model from the marked data; Based on a preset fourth data screening rule, a fourth data set that is annotated multiple times based on the preset annotation rule is screened out from the annotated data; based on a preset fifth data screening rule, a fifth data set that meets the error rate requirement of the preset annotation rule is screened out from the data to be annotated and the annotated data; based on a preset sixth data screening rule, a sixth data set that meets the preset proportional hit number requirement of the preset annotation rule is screened out from the data to be annotated and the annotated data; Based on the preset seventh data screening rule, screening the seventh data set whose number of label associations reaches the preset association threshold from the annotated data; The reference data set is determined according to the first data set, the second data set, the third data set, the fourth data set, the fifth data set, the sixth data set, and the seventh data set.
6. The classification model intelligent progressive training system according to claim 1, characterized in that: The reference data set determination module is also used to randomly extract a preset proportion of data from the reference data set to obtain a data sample set; the human-computer interaction module is also used to output the data sample set and obtain parameter adjustment instructions of the annotation rule model input by the user.
7. The classification model intelligent progressive training system according to claim 6, characterized in that: The reference data set determination module is also used to extract data features of the data sample set based on preset feature extraction rules, and to calculate statistical information of the data sample set based on preset data statistical rules; the human-computer interaction module is also used to output the data features and the statistical information, and obtain parameter adjustment instructions of the labeling rule model input by the user.
8. The classification model intelligent progressive training system according to claim 7, characterized in that: The human-computer interaction module is also used to send the data features and the statistical information to a designated terminal for display.
9. A classification model intelligent progressive training method, characterized in that: include: Obtain the data to be labeled, the preset initial deep learning model, and the preset labeling rule model; Based on a preset labeling rule, target data in the data to be labeled is labeled by using the labeling rule model to obtain labeled data; The initial deep learning model is trained by using the labeled data until a preset training end requirement is met to obtain a trained deep learning model; Based on the preset data screening rules, a reference data set is determined according to the data to be labeled, the labeled data and the preset labeling rules; the data screening rules are constructed based on the labeling rule model and the trained deep learning model; the reference data set is used to indicate the data set that does not conform to the preset labeling rules in the data to be labeled and the labeled data; Outputting the reference data set, and obtaining a parameter adjustment instruction of the annotation rule model input by a user; The parameters of the labeling rule model are adjusted according to the parameter adjustment instruction to obtain a classification model.
10. The classification model intelligent progressive training method according to claim 9, characterized in that: After the step of adjusting the parameters of the labeling rule model according to the parameter adjustment instruction to obtain the classification model, the method further includes: Get and update preset annotation rules; Based on the updated preset labeling rules, the updated target data in the data to be labeled is labeled through the labeling rule model to obtain updated labeled data; the trained deep learning model is retrained through the updated labeling data until the training end requirements are met to obtain an updated deep learning model.
Citation Information
Patent Citations
Multi-dimensional parameter statistics method and device for collection data
CN111400293A
Data labeling method and device
CN112269877A
Data labeling method based on cross validation and related equipment
CN112686301A
High-universality many-to-many relation triple extraction method fusing five-dimensional features
CN114091464A
Text classification method and apparatus, electronic device, and computer non-volatile readable storage medium
WO2021051560A1
