Model training method and device, electronic equipment, medium and program product

Abstract

Embodiments of the present specification disclose a model training method and device, electronic equipment, medium and program product. The method comprises: obtaining a first annotation dataset comprising original data and first annotation labels corresponding to the original data, and a training dataset obtained by filtering the first annotation dataset based on annotation rules satisfied by the original data corresponding to each first annotation label in the first annotation dataset; training an initial model based on the training dataset to obtain a first model; in a case where the first model does not meet preset requirements, annotating the original data based on the first model to obtain a second annotation dataset comprising the original data and second annotation labels corresponding to the original data, updating the first annotation dataset to the second annotation dataset, and performing again the steps of obtaining the first annotation dataset and the training dataset obtained by filtering the first annotation dataset based on the annotation rules until the first model meets the preset requirements, and the model training is completed.

Description

Technical Field

[0001] This specification relates to the field of computer technology, and in particular to a model training method, apparatus, electronic device, medium, and program product. Background Technology

[0002] With the application of artificial intelligence in various industries, model training often involves unsupervised learning, semi-supervised learning, and supervised learning. Because supervised learning trains models based on labeled data, it often outperforms unsupervised and semi-supervised learning models. Currently, the training process for supervised learning models typically involves first obtaining labeled training data through manual annotation, and then using this manually processed data to train the model. Summary of the Invention

[0003] This specification provides a model training method, apparatus, electronic device, medium, and program product. By leveraging the rigor of rules and the generalization ability of the model, and iterating the model and data labels multiple times, it reduces the professional requirements for data labelers, gradually improving the quality of the training dataset, and weakens the impact of data labeling quality on model performance. This allows for obtaining a higher-performing model through multiple iterations. The above technical solution is as follows:

[0004] Firstly, the embodiments of this specification provide a model training method, including:

[0005] Obtain a first labeled dataset and a training dataset obtained by filtering the first labeled dataset based on the labeling rules; the first labeled dataset includes the original data and the first label corresponding to the original data; the labeling rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset.

[0006] The initial model is trained based on the above training dataset to obtain the first model;

[0007] If the first model does not meet the preset requirements, the original data is labeled based on the first model to obtain a second labeled dataset; the second labeled dataset includes the original data and the second label corresponding to the original data.

[0008] The first labeled dataset is updated to the second labeled dataset, and the steps of obtaining the first labeled dataset and obtaining the training dataset obtained after filtering the first labeled dataset based on the labeling rules are executed again until the first model meets the preset requirements and the first model is determined as the target model that has been trained.

[0009] In one possible implementation, obtaining the first labeled dataset includes:

[0010] Obtain the first labeled dataset after labeling the original dataset based on a preset clustering algorithm.

[0011] In one possible implementation, after obtaining the first labeled dataset and before obtaining the training dataset obtained by filtering the first labeled dataset based on the labeling rules, the method further includes:

[0012] Obtain the target first labeled dataset after cleaning the first labeled dataset based on preset rules;

[0013] The training dataset obtained by filtering the first labeled dataset based on the annotation rules mentioned above includes:

[0014] Obtain the training dataset obtained after filtering the first labeled dataset of the above target based on the annotation rules; the annotation rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset of the above target.

[0015] In one possible implementation, the original data is labeled based on the first model to obtain a second labeled dataset, including:

[0016] The original data from the first labeled dataset is input into the first model, and the second labeled labels corresponding to the original data are output to obtain the second labeled dataset.

[0017] In one possible implementation, the original data is labeled based on the first model to obtain a second labeled dataset, including:

[0018] The original data in the first labeled dataset is input into the first model, and the predicted labeling results corresponding to the original data are output. The predicted labeling results include the predicted labeling labels corresponding to each of the original data and the labeling effect corresponding to the predicted labeling labels.

[0019] Based on the target conditions and the above prediction and annotation results, determine the second annotation label corresponding to the above original data;

[0020] The first annotation label corresponding to the original data is updated to the second annotation label to obtain the second annotation dataset.

[0021] In one possible implementation, determining the second label corresponding to the original data based on the target conditions and the predicted labeling results includes:

[0022] If the above annotation effect meets the above target conditions, the above predicted annotation label will be determined as the second annotation label corresponding to the above original data.

[0023] In one possible implementation, the aforementioned objective condition is negatively correlated with the performance metrics of the first model.

[0024] In one possible implementation, the above objective condition is the first condition when the performance index of the first model is less than the first index.

[0025] If the performance index of the first model is greater than or equal to the first index and less than the second index, the target condition is the second condition determined based on the first condition and the performance index.

[0026] If the performance index of the first model is greater than that of the second model, the above objective condition becomes the third condition.

[0027] Among them, the first indicator is lower than the second indicator; the first condition is higher than the second condition; and the second condition is higher than the third condition.

[0028] In one possible implementation, the original data is labeled based on the first model to obtain a second labeled dataset, including:

[0029] Based on the first annotation label and the performance index of the first model corresponding to the first annotation label, the second annotation label corresponding to the original data is determined, and the second annotation dataset is obtained.

[0030] Secondly, embodiments of this specification provide a model training apparatus, including:

[0031] The first acquisition module is used to acquire the first labeled dataset; the first labeled dataset includes the original data and the first label corresponding to the original data.

[0032] The second acquisition module is used to acquire the training dataset obtained after filtering the first labeled dataset based on the annotation rules; the annotation rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset.

[0033] The training module is used to train the initial model based on the above training dataset to obtain the first model;

[0034] The annotation module is used to annotate the original data based on the first model when the first model does not meet the preset requirements, to obtain a second annotated dataset; the second annotated dataset includes the original data and the second annotation labels corresponding to the original data.

[0035] The update module is used to update the first labeled dataset to the second labeled dataset.

[0036] The determination module is used to perform the above steps of analyzing the first labeled dataset to obtain the labeling rule base again until the first model meets the above preset requirements, and the first model is determined as the target model that has been trained; the second labeled dataset includes the above original data and the second label corresponding to the above original data.

[0037] In one possible implementation, the first acquisition module described above is specifically used for:

[0038] Obtain the first labeled dataset after labeling the original dataset based on a preset clustering algorithm.

[0039] In one possible implementation, the above-mentioned model training device further includes:

[0040] The third acquisition module is used to acquire the target first labeled dataset after data cleaning of the first labeled dataset based on preset rules.

[0041] The second acquisition module mentioned above is specifically used for:

[0042] Obtain the training dataset obtained after filtering the first labeled dataset of the above target based on the annotation rules; the annotation rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset of the above target.

[0043] In one possible implementation, the above annotation module is specifically used for:

[0044] The original data from the first labeled dataset is input into the first model, and the second labeled labels corresponding to the original data are output to obtain the second labeled dataset.

[0045] In one possible implementation, the above annotation module includes:

[0046] The prediction unit is used to input the original data in the first labeled dataset into the first model and output the predicted labeling results corresponding to the original data; the predicted labeling results include the predicted labeling labels corresponding to each of the original data and the labeling effect corresponding to the predicted labeling labels.

[0047] The determining unit is used to determine the second label corresponding to the original data based on the target conditions and the above-mentioned prediction labeling results;

[0048] The update unit is used to update the first annotation label corresponding to the original data to the second annotation label to obtain the second annotation dataset.

[0049] In one possible implementation, the aforementioned determining unit is specifically used for:

[0050] If the above annotation effect meets the above target conditions, the above predicted annotation label will be determined as the second annotation label corresponding to the above original data.

[0051] In one possible implementation, the aforementioned objective condition is negatively correlated with the performance metrics of the first model.

[0052] In one possible implementation, the above objective condition is the first condition when the performance index of the first model is less than the first index.

[0053] If the performance index of the first model is greater than or equal to the first index and less than the second index, the target condition is the second condition determined based on the first condition and the performance index.

[0054] If the performance index of the first model is greater than that of the second model, the above objective condition becomes the third condition.

[0055] Among them, the first indicator is lower than the second indicator; the first condition is higher than the second condition; and the second condition is higher than the third condition.

[0056] In one possible implementation, the above annotation module is specifically used for:

[0057] Based on the first annotation label and the performance index of the first model corresponding to the first annotation label, the second annotation label corresponding to the original data is determined, and the second annotation dataset is obtained.

[0058] Thirdly, embodiments of this specification provide an electronic device, including: a processor and a memory;

[0059] The processor is connected to the memory.

[0060] The aforementioned memory is used to store executable program code;

[0061] The processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to execute the method provided by the first aspect of the embodiments of this specification or any possible implementation of the first aspect.

[0062] Fourthly, embodiments of this specification provide a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method provided by the first aspect of the embodiments of this specification or any possible implementation thereof.

[0063] Fifthly, embodiments of this specification provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to execute the model training method provided by the first aspect or any possible implementation of the first aspect of the embodiments of this specification.

[0064] This embodiment first obtains a first labeled dataset and a training dataset obtained by filtering the first labeled dataset based on labeling rules. The first labeled dataset includes original data and first labeling labels corresponding to the original data. The labeling rules include data rules satisfied by the original data corresponding to each first labeling label in the first labeled dataset. Then, an initial model is trained based on the training dataset to obtain a first model. If the first model does not meet the preset requirements, the original data is labeled based on the first model to obtain a second labeled dataset. The second labeled dataset includes the original data and second labeling labels corresponding to the original data. Finally, the first labeled dataset is updated to the second labeled dataset, and the steps of obtaining the first labeled dataset and obtaining the training dataset obtained by filtering the first labeled dataset based on labeling rules are executed again until the first model meets the preset requirements, and the first model is determined as the target model that has completed training. This specification's embodiments leverage the rigor of rules and the generalizability of the model to iterate the model and data labels multiple times. Regardless of the quality of the annotations provided by the annotators, the quality of the training dataset used by the model gradually improves during iterations. This reduces the professional requirements for data annotators and weakens the impact of data annotation quality on model performance. Even if the initial labeled dataset is of low quality, a model with higher accuracy (performance) than the corresponding annotation accuracy of the initial labeled dataset can be trained through multiple iterations. In other words, the model performance no longer depends on the quality of the training data annotations in a single iteration, enabling the acquisition of a superior model through multiple iterations. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a schematic diagram illustrating the implementation process of model training in related technologies.

[0067] Figure 2A schematic diagram of the architecture of a model training system provided for an exemplary embodiment of this specification;

[0068] Figure 3 A schematic flowchart illustrating a model training method provided for an exemplary embodiment of this specification;

[0069] Figure 4 This is a schematic diagram illustrating an implementation process for obtaining a first labeled dataset, provided as an exemplary embodiment of this specification.

[0070] Figure 5 This is a schematic diagram illustrating an implementation process for obtaining a training dataset, provided as an exemplary embodiment of this specification.

[0071] Figure 6 This is a schematic diagram illustrating an implementation process for obtaining a second labeled dataset, provided as an exemplary embodiment of this specification.

[0072] Figure 7 A schematic diagram illustrating a target condition provided for an exemplary embodiment of this specification;

[0073] Figure 8 A schematic diagram illustrating the implementation process of a model training method provided in an exemplary embodiment of this specification;

[0074] Figure 9 A schematic diagram of the structure of a model training device provided for an exemplary embodiment of this specification;

[0075] Figure 10 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0076] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings.

[0077] The terms "first," "second," "third," etc., used in this specification, claims, and the foregoing drawings are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such processes, methods, products, or apparatus.

[0078] like Figure 1As shown, in related model training techniques, the process often involves first manually labeling each data point in the original dataset to obtain a labeled dataset. This manually labeled dataset is then used as the labeled training dataset for model training, resulting in the trained target model. However, the performance of the target model trained in this way is overly dependent on the quality of the labeled dataset. For some complex tasks, it is difficult for annotators to control the labeling quality, often leading to low-quality labeled data and consequently, poor performance of the trained target model.

[0079] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the architecture of a model training system provided for an exemplary embodiment of this specification. Figure 2 As shown, the model training system may include: a terminal 210 and a server 220. Wherein:

[0080] Terminal 210 may include one or more user terminals. User-version software can be installed on terminal 210 to enable users to annotate the original dataset online to obtain a first labeled dataset. The first labeled dataset includes the original data and the corresponding first label labels. The software also includes functions such as summarizing and analyzing the first labeled dataset to obtain corresponding annotation rules, which include data rules satisfied by the original data corresponding to each first label label in the first labeled dataset. Terminal 210 can establish a data relationship with the network and, through this network, establish a data connection with server 220, for example, by sending the first labeled dataset and annotation rules.

[0081] Understandably, any user terminal in terminal 210 can be, but is not limited to, a mobile phone, tablet computer, laptop computer, or other device with user software installed.

[0082] Server 220 can be a server capable of providing training for multiple models. It can receive data such as the first labeled dataset and the labeling rule set sent by terminal 210 via the network. Server 220 can also first obtain the first labeled dataset and the training dataset obtained after filtering the first labeled dataset based on the labeling rules. Then, it can train the initial model based on the training dataset to obtain the first model. If the first model does not meet the preset requirements, it can label the original data based on the first model to obtain the second labeled dataset. The second labeled dataset includes the original data and the second label corresponding to the original data. Finally, it updates the first labeled dataset to the second labeled dataset and repeats the steps of obtaining the first labeled dataset and the training dataset obtained after filtering the first labeled dataset based on the labeling rules until the first model meets the preset requirements. The first model is then determined as the target model that has been trained.

[0083] Understandably, server 220 can be, but is not limited to, a hardware server, a virtual server, a cloud server, etc.

[0084] The model training described above is not limited to execution by server 220; it can also be executed by any user terminal in terminal 210. The user terminal can obtain the first labeled dataset and the training dataset obtained after filtering the first labeled dataset based on labeling rules through the user-version software installed on it. Then, it trains the initial model based on the training dataset to obtain the first model. If the first model does not meet the preset requirements, it labels the original data based on the first model to obtain the second labeled dataset, which includes the original data and the corresponding second label labels. Finally, the first labeled dataset is updated to the second labeled dataset, and the step of obtaining the training dataset obtained after filtering the first labeled dataset based on labeling rules is executed again until the first model meets the preset requirements, at which point the first model is determined as the target model that has completed training. This specification does not specifically limit this embodiment; all subsequent embodiments use server 220 as an example for illustrating the model training.

[0085] The network can be a medium that provides a communication link between server 220 and any terminal 210, or it can be the Internet, which includes network devices and transmission media, and is not limited thereto. The transmission media can be a wired link (e.g., but not limited to, coaxial cable, fiber optic cable, and digital subscriber line (DSL)) or a wireless link (e.g., but not limited to, wireless fidelity (WIFI), Bluetooth, and mobile device networks).

[0086] Understandably, Figure 2 The number of terminals 210 and servers 220 in the model training system shown is merely an example. In a specific implementation, the model training system can contain any number of terminals and servers, and this specification does not specifically limit this. For example, but not limited to, terminal 210 can be a terminal cluster composed of multiple terminals, and server 220 can be a server cluster composed of multiple servers.

[0087] Next, combine Figure 2 Taking model training performed by server 220 as an example, this document describes the model training method provided in the embodiments. Please refer to [link / reference needed] for details. Figure 3 This is a flowchart illustrating a model training method provided in an exemplary embodiment of this specification. Figure 3 As shown, the model training method includes the following steps:

[0088] S302, obtain the first labeled dataset and the training dataset obtained after filtering the first labeled dataset based on the labeling rules. The labeling rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset.

[0089] Specifically, the first labeled dataset includes the original data and the corresponding first labeled tags. The original dataset may be, but is not limited to, text data, audio data, image data, or video data.

[0090] Optionally, server 220 may first receive the original dataset sent by network terminal 210 or other servers, and then label the original dataset based on a preset clustering algorithm to obtain a first labeled dataset. Alternatively, server 220 may directly receive the first labeled dataset obtained after labeling the original dataset based on a preset clustering algorithm sent by network terminal 210. This embodiment of the specification does not limit this approach. The preset clustering algorithms mentioned above include, but are not limited to, K-means clustering algorithm, Fuzzy-c Means (FCM) clustering algorithm, etc.

[0091] like Figure 4 As shown, the process of annotating the original dataset using a preset clustering algorithm to obtain the first labeled dataset can be as follows: first, the original dataset is clustered using the preset clustering algorithm, that is, the original dataset is divided into multiple categories, and then the original data in each category is labeled to obtain the first labeled dataset, that is, the original data in one category corresponds to the same first label. The above-mentioned labeling of the original data in each category can be achieved by labelers labeling the original data in each category, or it can be done without human labeling, but by first identifying the keywords corresponding to the original data in each category, and then directly determining the keywords as the first label corresponding to the original data in each category, etc. The embodiments in this specification do not limit this.

[0092] In the embodiments described in this specification, the original dataset is first clustered using a clustering algorithm. Then, only the original data under each category needs to be uniformly labeled to obtain the first labeled dataset. This avoids the process of labeling each original data in the original dataset one by one, solves the cold start problem, reduces the professional requirements for labeling personnel and reduces the consumption of human resources, and improves the labeling efficiency of the original dataset, thereby improving the efficiency of model training.

[0093] Optionally, such as Figure 5As shown, the above-described process of obtaining the training dataset after filtering the first labeled dataset based on annotation rules can involve first analyzing the first labeled dataset to obtain the annotation rules satisfied by most of the original data corresponding to each first labeled label in the first labeled dataset, and then using the obtained annotation rules to filter out the original data in the first labeled dataset that do not satisfy the annotation rules, thereby obtaining the training dataset. All the original data corresponding to each first labeled label in the above-described training dataset satisfy the annotation rules. The above-described data analysis of the first labeled dataset to obtain the annotation rules satisfied by most of the original data corresponding to each first labeled label in the first labeled dataset can be performed by annotation personnel analyzing the data according to scenario and algorithm requirements, summarizing relevant data rules, or it can be that the server 220 directly extracts the conditions (annotation rules) satisfied by most of the original data corresponding to each first labeled label in the first labeled dataset. This embodiment of the specification does not limit this. The number of annotation rules satisfied by most of the original data corresponding to each first labeled label can be one or more, and this embodiment of the specification does not limit this. The above annotation rules may include, but are not limited to, the original data corresponding to the first annotation label including a certain keyword and / or the size of the original data corresponding to the first annotation label being less than the target size, etc.

[0094] For example, if the analysis reveals that the original data corresponding to the first label A in the first annotation dataset satisfies rule a and rule b, and the analysis reveals that the original data corresponding to the first label B in the first annotation dataset satisfies rule b, and only 80% of the original data corresponding to the first label A simultaneously satisfies rule a and rule b, and only 70% of the original data corresponding to the first label B satisfies rule b, then based on rules a and b, 20% of the original data corresponding to the first label A that does not simultaneously satisfy rules a and b can be filtered out, and based on rule b, 30% of the original data corresponding to the first label B that does not satisfy rule b can be filtered out. This results in training data where all original data corresponding to the first label satisfies the corresponding annotation rules. This embodiment of the specification can filter out original data in the first annotation dataset that does not satisfy the annotation rules under the corresponding first label through the rigor of the rules, thereby obtaining a training dataset with higher annotation quality than the first annotation dataset, leading to higher performance of the model trained on the training dataset.

[0095] Understandably, during multiple training iterations, as the quality of the annotations in the first annotation dataset continuously improves, the data rules, i.e., the annotation rules, that the original data corresponding to each first annotation label in the first annotation dataset satisfy become more complete, and their coverage of the original data under that first annotation label shows an increasing trend. As the annotation rules become more complete and the coverage increases, the annotation quality of the training dataset obtained after filtering the first annotation dataset based on the annotation rules also continuously improves, thereby ensuring that the performance of the first model trained on the training dataset will also improve.

[0096] Optionally, to avoid the presence of outliers in the first labeled dataset affecting the accuracy of the annotation rules or the usability of the training dataset, thus leading to lower performance of the trained model, after obtaining the first labeled dataset and before obtaining the training dataset obtained by filtering the first labeled dataset based on the annotation rules, a target first labeled dataset can be obtained after data cleaning of the first labeled dataset based on preset rules. Then, the training dataset obtained after filtering the target first labeled dataset based on the annotation rules is obtained. The annotation rules include the data rules satisfied by the original data corresponding to each first label in the target first labeled dataset. The aforementioned data cleaning includes, but is not limited to, outlier handling, long-tail data cleaning, data resampling, and default value imputation, among other conventional data cleaning methods.

[0097] S304, the initial model is trained based on the training dataset to obtain the first model.

[0098] Specifically, the initial model mentioned above can be an untrained neural network model, or a trained model whose performance does not meet the preset requirements. This specification does not limit this in the embodiments. The preset requirements may include, but are not limited to, the accuracy of the model's output being greater than a preset accuracy threshold or the model's output meeting the requirements.

[0099] For example, in a classification scenario, the first model can be a classification model, and the first label corresponding to the original data in the training dataset can represent the category corresponding to the original data; in a generation scenario, the first model can be a generation model, and the first label corresponding to the original data in the training dataset can represent the generation result corresponding to the original data.

[0100] S306, Determine whether the first model meets the preset requirements.

[0101] Specifically, after the server 220 trains the first model using the training dataset, it can first determine whether the performance of the first model meets the preset requirements. If it does, it executes S312; if it does not, it executes S308.

[0102] For example, in a classification scenario, the aforementioned preset requirement may include, but is not limited to, the accuracy of the classification result output by the first model being greater than a preset accuracy threshold; in a question-answering scenario, the aforementioned preset requirement may include, but is not limited to, the answer output by the first model being relevant to the input question, i.e., the requirement, etc.

[0103] S308. If the first model does not meet the preset requirements, the original data is labeled based on the first model to obtain the second labeled dataset.

[0104] Specifically, the second labeled dataset includes the original data and the corresponding second labeled labels.

[0105] Optionally, the aforementioned raw data can be raw data from an unlabeled raw dataset, or raw data from a target first labeled dataset obtained after data cleaning of the first labeled dataset based on preset rules, etc. The embodiments in this specification do not limit this.

[0106] Optionally, if the first model does not meet the preset requirements, the original data in the first labeled dataset can be input into the first model, and the second labeled labels corresponding to the original data can be directly output to obtain the second labeled dataset; the second labeled labels can be, but are not limited to, the labels with the highest probability or the best labeling effect corresponding to the original data output by the first model.

[0107] S310, update the first labeled dataset to the second labeled dataset.

[0108] Specifically, if the first model does not meet the preset requirements, the original data is labeled based on the first model to obtain the second labeled dataset. Then, the first labeled dataset can be updated to the second labeled dataset, and S302 is executed again to obtain the first labeled dataset and the training dataset obtained after filtering the first labeled dataset based on the labeling rules. This process continues until the first model meets the preset requirements. Then, S312 is executed, which indicates that the training of the first model has reached the target and the first model can be identified as the target model that has completed training.

[0109] S312, if the first model meets the preset requirements, the first model is determined as the target model that has been trained.

[0110] This specification's embodiments leverage the rigor of rules and the generalizability of the model to iterate the model and data labels multiple times. Regardless of the quality of the annotations provided by the annotators, the quality of the training dataset used by the model gradually improves during iterations. This reduces the professional requirements for data annotators and weakens the impact of data annotation quality on model performance. Even if the initial labeled dataset is of low quality, a model with higher accuracy (performance) than the corresponding annotation accuracy of the initial labeled dataset can be trained through multiple iterations. In other words, the model performance no longer depends on the quality of the training data annotations in a single iteration, enabling the acquisition of a superior model through multiple iterations.

[0111] Optionally, to ensure that the annotation quality of the second annotation dataset is higher than that of the first annotation dataset, and to avoid the problem that a lower performance of the first model leads to a lower annotation quality in the second training dataset, such as... Figure 6 As shown, in step S308 above, when the first model does not meet the preset requirements, the process of annotating the original data based on the first model to obtain the second labeled dataset may include the following steps:

[0112] S602, input the original data from the first labeled dataset into the first model, and output the predicted labeled results corresponding to the original data.

[0113] Specifically, the prediction labeling results include the predicted labeling labels corresponding to each original data point and the labeling effect corresponding to the predicted labeling labels.

[0114] Understandably, the number of predicted labels corresponding to each piece of raw data may include one or more, and this specification does not limit this. In classification scenarios, the above-mentioned labeling effect can be the probability corresponding to the predicted label; in generation scenarios, the above-mentioned labeling effect can be the generation effect corresponding to the predicted label, such as, but not limited to, whether the predicted label meets the generation requirements or the accuracy of the predicted label.

[0115] S604, determine the second annotation label corresponding to the original data based on the target conditions and the predicted annotation results.

[0116] Optionally, after the first model outputs the predicted labeling results corresponding to the original data, if the labeling effect meets the target conditions, the predicted labeling label can be determined as the second labeling label corresponding to the original data. That is, first determine whether the labeling effect corresponding to each predicted labeling label in the predicted labeling results meets the target conditions, and then determine the predicted labeling label that meets the target conditions as the second labeling label corresponding to the original data.

[0117] For example, when the annotation effect is the probability corresponding to the predicted annotation label, the objective condition can be that the probability corresponding to the predicted annotation label is greater than a preset probability threshold; when the annotation effect is the generated effect corresponding to the predicted annotation label, the objective condition can be that the generated effect corresponding to the predicted annotation label meets the preset generation requirements.

[0118] Optionally, the objective condition is negatively correlated with the performance metric of the first model. That is, if the first model does not meet the preset requirements, as the labeled dataset is continuously updated, the performance of the first model trained will become increasingly better. When the performance of the first model is poor, in order to ensure the labeling quality of the second labeled dataset obtained by labeling the original data based on the first model, and to ensure that the performance of the first model trained in the next iteration is better, the objective condition corresponding to the first model should be relatively strict (high). Conversely, as the performance of the first model gradually improves, the objective condition corresponding to the first model can also be gradually reduced, thereby ensuring that the second labeled labels corresponding to each original data in the second labeled dataset have a higher labeling accuracy and better labeling effect compared to the first labeled labels.

[0119] Optionally, to avoid the problem of overly strict (high) target conditions leading to failed annotation of the original data based on the first model when the target condition is negatively correlated with the performance index of the first model, or the impact of overly low target conditions on the accuracy of annotation of the original data based on the first model when the performance index of the first model is lower than the first index, the target condition can be the first condition if the performance index of the first model is lower than the first index; the target condition can be the second condition determined based on the first condition and the performance index if the performance index of the first model is greater than or equal to the first index and less than the second index; and the target condition can be the third condition if the performance index of the first model is greater than the second index. Wherein, the first index is lower than the second index; the first condition is higher than the second condition; and the second condition is higher than the third condition. The aforementioned performance index can be, but is not limited to, the accuracy, effectiveness, etc., corresponding to the output results of the first model.

[0120] For example, such as Figure 7 As shown, when the performance index 'a' of the first model is less than or equal to 'a1', the objective condition 'b' can be set to 'b1'. When the performance index 'a' of the first model is 'a2', which is greater than 'a1' and less than 'a3', the objective condition 'b' can be set to 'b2', where 'b2' is less than 'b1' and greater than 'b3'. When the performance index 'a' of the first model is greater than or equal to 'a3', the objective condition 'b' can be set to 'b3'. Where 'b2' = -k × 'a2' + 'b1', and 'k' is a preset value.

[0121] S606, update the first annotation label corresponding to the original data to the second annotation label to obtain the second annotation dataset.

[0122] Specifically, after determining the second label corresponding to the original data, the first label corresponding to the original dataset can be directly updated to the second label to obtain the second labeled dataset.

[0123] Understandably, since the original data labeled by the first model may be the original data in the unlabeled original dataset, or it may be the original data in the target first labeled dataset obtained after cleaning the first labeled dataset according to preset rules, the original data in the second labeled dataset may include not only the original data in the target first labeled dataset, but also the original data in the original dataset that has not been cleaned. This allows the training dataset to be generalized and expanded to a certain extent during the iteration process, increasing the generalization ability of the first model.

[0124] Optionally, the implementation process of S308 above can also be based on determining the second label corresponding to the original data based on the first label and the performance index of the first model corresponding to the first label, thus obtaining the second labeled dataset. That is, the first labeled labels corresponding to the original data can be fused according to the performance index of the first model obtained in each training, thereby obtaining the second labeled labels corresponding to the original data, and the second labeled dataset including the original data and the second labeled labels corresponding to the original data.

[0125] For example, if the first annotation labels corresponding to the original data are k1, k2, ..., k n The performance metrics of the first model obtained from the training are a1, a2, ..., a n Then the second label k corresponding to the original data n+1 It can be:

[0126] k n+1 = k1*a1 + k2*a2 + ... + k n *a n ;or or

[0127] k n+1 = b1k1*a1 + b2k2*a2 + ... + b n k n *a n ;or Where e is the natural logarithm; b1, b2, ..., b n Here, n represents the objective condition corresponding to the first model, and n is a positive integer.

[0128] Next, combine Figures 3-7 This specification describes the implementation process of a model training method provided in an exemplary embodiment, and details how... Figure 8As shown: First, an unlabeled raw dataset can be collected. Then, the raw dataset is clustered and labeled. Based on the labeled raw dataset, abnormal raw data is cleaned to obtain a first labeled dataset. Then, based on the raw data corresponding to each first label in the first labeled dataset, the labeling rules that the raw data corresponding to each first label may need to meet can be analyzed. Based on the labeling rules, the raw data under each first label in the first labeled dataset that does not meet the corresponding labeling rules can be filtered out to obtain a training dataset in which the labeling rules can completely cover the raw data under the corresponding first label. Then, a first model is trained based on this training dataset. At this time, if the first model does not meet the preset requirements, the raw data needs to be labeled based on the first model to obtain a second labeled dataset. Then, the first labeled dataset is updated to the second labeled dataset, and data analysis is performed again to update the more complete labeling rules corresponding to the new first labeled dataset, as well as updating the training dataset, etc., until the trained first model meets the preset requirements. Then, the model training can be determined to be complete, and the first model can be identified as the target model. The model training method provided in the embodiments of this specification can be used to complete model training. The model performance is not limited by the labeling quality of a single training dataset. To obtain a high-performance model, iterative training can be performed continuously. At the same time, after training is completed, not only can a high-performance model be obtained, but also a training dataset with high labeling quality and labeling rules with strong reliability and wider coverage.

[0129] Please refer to Figure 9 This is a schematic diagram of the structure of a model training device provided in an exemplary embodiment of this specification. Figure 9 As shown, the model training device 900 includes:

[0130] The first acquisition module 910 is used to acquire a first labeled dataset; the first labeled dataset includes raw data and the first labeled labels corresponding to the raw data.

[0131] The second acquisition module 920 is used to acquire the training dataset obtained after filtering the first labeled dataset based on the annotation rules; the annotation rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset.

[0132] Training module 930 is used to train the initial model based on the above training dataset to obtain the first model;

[0133] The annotation module 940 is used to annotate the original data based on the first model when the first model does not meet the preset requirements, to obtain a second annotated dataset; the second annotated dataset includes the original data and the second annotation labels corresponding to the original data.

[0134] Update module 950 is used to update the first labeled dataset to the second labeled dataset.

[0135] The determination module 960 is used to perform the above steps of data analysis on the first labeled dataset to obtain the labeling rule base again until the first model meets the above preset requirements, and the first model is determined as the target model that has been trained; the second labeled dataset includes the above original data and the second label corresponding to the above original data.

[0136] In one possible implementation, the first acquisition module 910 is specifically used for:

[0137] Obtain the first labeled dataset after labeling the original dataset based on a preset clustering algorithm.

[0138] In one possible implementation, the model training device 900 further includes:

[0139] The third acquisition module is used to acquire the target first labeled dataset after data cleaning of the first labeled dataset based on preset rules.

[0140] The second acquisition module 920 mentioned above is specifically used for:

[0141] Obtain the training dataset obtained after filtering the first labeled dataset of the above target based on the annotation rules; the annotation rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset of the above target.

[0142] In one possible implementation, the annotation module 940 is specifically used for:

[0143] The original data from the first labeled dataset is input into the first model, and the second labeled labels corresponding to the original data are output to obtain the second labeled dataset.

[0144] In one possible implementation, the annotation module 940 includes:

[0145] The prediction unit is used to input the original data in the first labeled dataset into the first model and output the predicted labeling results corresponding to the original data; the predicted labeling results include the predicted labeling labels corresponding to each of the original data and the labeling effect corresponding to the predicted labeling labels.

[0146] The determining unit is used to determine the second label corresponding to the original data based on the target conditions and the above-mentioned prediction labeling results;

[0147] The update unit is used to update the first annotation label corresponding to the original data to the second annotation label to obtain the second annotation dataset.

[0148] In one possible implementation, the aforementioned determining unit is specifically used for:

[0149] If the above annotation effect meets the above target conditions, the above predicted annotation label will be determined as the second annotation label corresponding to the above original data.

[0150] In one possible implementation, the aforementioned objective condition is negatively correlated with the performance metrics of the first model.

[0151] In one possible implementation, the above objective condition is the first condition when the performance index of the first model is less than the first index.

[0152] If the performance index of the first model is greater than or equal to the first index and less than the second index, the target condition is the second condition determined based on the first condition and the performance index.

[0153] If the performance index of the first model is greater than that of the second model, the above objective condition becomes the third condition.

[0154] Among them, the first indicator is lower than the second indicator; the first condition is higher than the second condition; and the second condition is higher than the third condition.

[0155] In one possible implementation, the annotation module 940 is specifically used for:

[0156] Based on the first annotation label and the performance index of the first model corresponding to the first annotation label, the second annotation label corresponding to the original data is determined, and the second annotation dataset is obtained.

[0157] The division of modules in the above-described model training device is for illustrative purposes only. In other embodiments, the model training device can be divided into different modules as needed to complete all or part of the functions of the above-described model training device. The implementation of each module in the model training device provided in the embodiments of this specification can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the model training method described in the embodiments of this specification.

[0158] Please see Figure 10 , Figure 10 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. For example... Figure 10As shown, the electronic device 1000 may include: at least one processor 1010, at least one communication bus 1020, a user interface 1030, at least one network interface 1040, and a memory 1050. The communication bus 1020 can be used to enable communication between the aforementioned components.

[0159] The user interface 1030 may include a display screen and a camera. Optional user interfaces may also include standard wired interfaces and wireless interfaces.

[0160] The network interface 1040 may optionally include a Bluetooth module, a Near Field Communication (NFC) module, a Wireless Fidelity (Wi-Fi) module, etc.

[0161] The processor 1010 may include one or more processing cores. The processor 1010 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data of the routing electronic device 1000 by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1050, and by calling data stored in the memory 1050. Optionally, the processor 1010 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1010 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip without being integrated into the processor 1010.

[0162] The memory 1050 may include random access memory (RAM) or read-only memory (ROM). Optionally, the memory 1050 may include a non-transitory computer-readable medium. The memory 1050 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1050 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a fetch function, a training function, an update function, etc.), instructions for implementing the various method embodiments described above, etc.; the data storage area may store data involved in the various method embodiments described above, etc. Optionally, the memory 1050 may also be at least one storage device located remotely from the aforementioned processor 1010. Figure 10 As shown, the memory 1050, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and program instructions.

[0163] Specifically, the processor 1010 can be used to call program instructions stored in the memory 1050 and perform the following operations:

[0164] Obtain a first labeled dataset and a training dataset obtained by filtering the first labeled dataset based on the labeling rules; the first labeled dataset includes the original data and the first label corresponding to the original data; the labeling rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset.

[0165] The initial model is trained based on the above training dataset to obtain the first model.

[0166] If the first model does not meet the preset requirements, the original data is labeled based on the first model to obtain a second labeled dataset; the second labeled dataset includes the original data and the second labeled labels corresponding to the original data.

[0167] The first labeled dataset is updated to the second labeled dataset, and the steps of obtaining the first labeled dataset and obtaining the training dataset obtained after filtering the first labeled dataset based on the labeling rules are executed again until the first model meets the preset requirements and the first model is determined as the target model that has been trained.

[0168] In some possible embodiments, when the processor 1010 performs the above-described acquisition of the first labeled dataset, it is specifically used to perform:

[0169] Obtain the first labeled dataset after labeling the original dataset based on a preset clustering algorithm.

[0170] In some possible embodiments, after the processor 1010 performs the above-described acquisition of the first labeled dataset and before the above-described acquisition of the training dataset obtained by filtering the first labeled dataset based on the labeling rules, it is further configured to perform:

[0171] Obtain the target first labeled dataset after cleaning the first labeled dataset based on preset rules.

[0172] When the processor 1010 executes the above-mentioned process of obtaining the training dataset after filtering the first labeled dataset based on the annotation rules, it is specifically used to perform the following:

[0173] Obtain the training dataset obtained after filtering the first labeled dataset of the above target based on the annotation rules; the annotation rules include the data rules satisfied by the original data corresponding to each first label in the first labeled dataset of the above target.

[0174] In some possible embodiments, when the processor 1010 performs the above-described annotation of the original data based on the first model to obtain the second labeled dataset, it specifically performs the following:

[0175] The original data from the first labeled dataset is input into the first model, and the second labeled labels corresponding to the original data are output to obtain the second labeled dataset.

[0176] In some possible embodiments, when the processor 1010 performs the above-described annotation of the original data based on the first model to obtain the second labeled dataset, it specifically performs the following:

[0177] The original data in the first labeled dataset is input into the first model, and the predicted labeling results corresponding to the original data are output. The predicted labeling results include the predicted labeling labels corresponding to each of the original data and the labeling effect corresponding to the predicted labeling labels.

[0178] Based on the target conditions and the above prediction and annotation results, the second annotation label corresponding to the above original data is determined.

[0179] The first annotation label corresponding to the original data is updated to the second annotation label to obtain the second annotation dataset.

[0180] In some possible embodiments, when the processor 1010 executes the above-described method of determining the second label corresponding to the original data based on the target conditions and the above-described prediction labeling results, it is specifically used to perform:

[0181] If the above annotation effect meets the above target conditions, the above predicted annotation label will be determined as the second annotation label corresponding to the above original data.

[0182] In some possible embodiments, the above target conditions are negatively correlated with the performance metrics of the first model described above.

[0183] In some possible embodiments, when the performance index of the first model is less than the first index, the target condition is the first condition.

[0184] If the performance index of the first model is greater than or equal to the first index and less than the second index, the target condition is the second condition determined based on the first condition and the performance index.

[0185] If the performance index of the first model is greater than that of the second model, the above objective condition becomes the third condition.

[0186] Among them, the first indicator is lower than the second indicator; the first condition is higher than the second condition; and the second condition is higher than the third condition.

[0187] In some possible embodiments, when the processor 1010 performs the above-described annotation of the original data based on the first model to obtain the second labeled dataset, it specifically performs the following:

[0188] Based on the first annotation label and the performance index of the first model corresponding to the first annotation label, the second annotation label corresponding to the original data is determined, and the second annotation dataset is obtained.

[0189] This specification also provides a computer-readable storage medium storing instructions that, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described model training apparatus are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium.

[0190] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this specification are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0191] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0192] The embodiments described above are merely preferred embodiments of this specification and are not intended to limit the scope of this specification. Any modifications and improvements made by those skilled in the art to the technical solutions of this specification without departing from the spirit of this specification should fall within the protection scope defined by the claims.

[0193] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims and specification may be performed in a different order than in the embodiments described in the specification and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. A model training method, the method comprising: Obtain the first labeled dataset, and obtain the training dataset obtained by filtering the first labeled dataset based on the labeling rules; The first labeled dataset includes raw data and the first labeled data corresponding to the raw data; The annotation rules include the data rules satisfied by the original data corresponding to each first annotation label in the first annotation dataset; the data rules are obtained based on data summary and analysis of the original data corresponding to each first annotation label in the first annotation dataset, and are used to describe the common characteristics of the original data under each first annotation label; the original data includes at least one of the following: text data, voice data, image data, and video data; The initial model is trained based on the training dataset to obtain the first model; If the first model does not meet the preset requirements, the original data is labeled based on the first model to obtain a second labeled dataset; the second labeled dataset includes the original data and the second label corresponding to the original data. The first labeled dataset is updated to the second labeled dataset, and the steps of obtaining the first labeled dataset and obtaining the training dataset obtained after filtering the first labeled dataset based on the labeling rules are executed again until the first model meets the preset requirements, and the first model is determined as the target model that has been trained. The step of labeling the original data based on the first model to obtain a second labeled dataset includes: inputting the original data in the first labeled dataset into the first model and outputting the predicted labeling results corresponding to the original data; the predicted labeling results include the predicted labeling labels corresponding to each of the original data and the labeling effect corresponding to the predicted labeling labels; determining the second labeling labels corresponding to the original data based on the target conditions and the predicted labeling results; updating the first labeling labels corresponding to the original data to the second labeling labels to obtain the second labeled dataset, wherein the target conditions are negatively correlated with the performance index of the first model.

2. The method as described in claim 1, wherein obtaining the first labeled dataset includes: Obtain the first labeled dataset after labeling the original dataset based on a preset clustering algorithm.

3. The method as described in claim 1, wherein after obtaining the first labeled dataset and before obtaining the training dataset obtained by filtering the first labeled dataset based on the labeling rules, the method further comprises: Obtain the target first labeled dataset after cleaning the first labeled dataset based on preset rules; The step of obtaining the training dataset after filtering the first labeled dataset based on the annotation rules includes: Obtain the training dataset obtained by filtering the first labeled dataset of the target based on the labeling rules; The annotation rules include the data rules satisfied by the original data corresponding to each first annotation label in the target first annotation dataset.

4. The method as described in claim 1, wherein labeling the original data based on the first model to obtain a second labeled dataset includes: The original data from the first labeled dataset is input into the first model, and the second labeled labels corresponding to the original data are output to obtain the second labeled dataset.

5. The method of claim 1, which determines the second label corresponding to the original data based on the target conditions and the predicted labeling result, includes: If the annotation effect meets the target conditions, the predicted annotation label is determined as the second annotation label corresponding to the original data.

6. The method as described in claim 1, wherein when the performance index of the first model is less than the first index, the target condition is the first condition; If the performance index of the first model is greater than or equal to the first index and less than the second index, the target condition is the second condition determined based on the first condition and the performance index. If the performance index of the first model is greater than that of the second model, the target condition is the third condition. Wherein, the first indicator is lower than the second indicator; the first condition is higher than the second condition; and the second condition is higher than the third condition.

7. The method as described in claim 1, wherein labeling the original data based on the first model to obtain a second labeled dataset comprises: Based on the first annotation label and the performance index of the first model corresponding to the first annotation label, the second annotation label corresponding to the original data is determined to obtain the second annotation dataset.

8. A model training apparatus, the apparatus comprising: The first acquisition module is used to acquire the first labeled dataset; The first labeled dataset includes raw data and the first labeled data corresponding to the raw data; The raw data includes at least one of the following: text data, voice data, image data, and video data; The second acquisition module is used to acquire the training dataset obtained after filtering the first labeled dataset based on the labeling rules; The annotation rules include the data rules satisfied by the original data corresponding to each first annotation label in the first annotation dataset; the data rules are obtained by summarizing and analyzing the original data corresponding to each first annotation label in the first annotation dataset, and are used to describe the common characteristics of the original data under each first annotation label; The training module is used to train the initial model based on the training dataset to obtain the first model; The annotation module is used to annotate the original data based on the first model to obtain a second annotated dataset when the first model does not meet the preset requirements. The update module is used to update the first labeled dataset to the second labeled dataset; The determination module is used to re-execute the step of performing data analysis on the first labeled dataset to obtain the labeling rule base until the first model meets the preset requirements, and to determine the first model as the target model that has completed training; the second labeled dataset includes the original data and the second label corresponding to the original data. The step of labeling the original data based on the first model to obtain a second labeled dataset includes: inputting the original data in the first labeled dataset into the first model and outputting the predicted labeling results corresponding to the original data; the predicted labeling results include the predicted labeling labels corresponding to each of the original data and the labeling effect corresponding to the predicted labeling labels; determining the second labeling labels corresponding to the original data based on the target conditions and the predicted labeling results; updating the first labeling labels corresponding to the original data to the second labeling labels to obtain the second labeled dataset, wherein the target conditions are negatively correlated with the performance index of the first model.

9. An electronic device, comprising: Processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-7.

10. A computer storage medium storing a plurality of instructions adapted for loading by a processor and performing the method steps of any one of claims 1-7.

11. A computer program product comprising instructions that, when run on a computer or processor, causes the computer or processor to perform the model training method as described in any one of claims 1-7.

Images