Method of expanding a data set and predicting a class of an object to be classified

Abstract

The application provides an extended training data set, and a method for classification using a classification model trained by the extended training data set. The method for extending the training data set includes generating merged data based on original data and auxiliary data, the auxiliary data being associated with an application scenario of the classification model and coming from a source different from a source of the original data; and updating a category of the merged data using the classification model to generate a first extended data set for training the classification model. The scheme of the application can expand the number of labeled data samples and improve the labeling accuracy of the data, so that the classification performance of the trained classification model is optimized.

Description

Technical Field

[0001] This application relates to data processing, and more particularly to methods, apparatus, and computer storage media for classifying objects using a classification model trained on an extended training dataset. Background Technology

[0002] In tasks that use classification models, such as neural network models, to predict object categories, labeled data is required to train the classification model. Therefore, the amount of data in the training dataset and the accuracy of the labeling are important factors affecting model performance.

[0003] The data in the training dataset typically comes from historical data samples of the objects to be classified in the application scenario of the classification task. However, using only labeled historical data of the objects to be classified may result in insufficient information to extract for determining model parameters. In addition, when the categories (also known as labels) of the objects to be classified include long-tail label types, the imbalance in the amount of historical data samples will mean that the historical data samples used to train the classification model are insufficient to cover all long-tail label types.

[0004] Therefore, there is a need to expand the training dataset to improve the training effect of the classification model and thus improve the classification performance of the classification model. Summary of the Invention

[0005] In view of the problems and defects mentioned above, embodiments of this application propose a method for expanding a training dataset, a method for classifying objects to be classified based on a classification model trained on the expanded dataset, and the device and computer storage medium thereof.

[0006] According to one aspect of this application, a method for expanding a training dataset comprising multiple original data sets is proposed, the method comprising:

[0007] Merged data is generated based on the original data and auxiliary data. The auxiliary data is related to the application scenario of the classification model and comes from a different source than the original data.

[0008] The classification model is used to update the categories of the merged data to generate the first expanded dataset for training the classification model.

[0009] According to embodiments of this application, updating the categories of merged data using a classification model to generate a first expanded dataset for training the classification model includes the following steps:

[0010] The merged data is divided into a first merged data subset and a second merged data subset that have different merged data.

[0011] The original classification model is trained using the first merged subset of data to obtain the first classification model;

[0012] The categories of the merged data in the second merged data subset are updated using the predicted categories of the merged data in the second merged data subset determined by the first classification model.

[0013] A first expanded dataset is generated based on a first merged subset of data and an updated second merged subset of data, wherein the first expanded dataset includes merged data with updated categories.

[0014] According to another aspect of this application, a method for predicting the category of an object to be classified is proposed, comprising: obtaining the object to be classified; and predicting the category of the object to be classified using a second classification model trained using the method described above.

[0015] According to another aspect of this application, a computer-readable storage medium is provided that stores a computer program thereon, the computer program including executable instructions that, when executed by a processor, implement the method described above.

[0016] According to another aspect of this application, an electronic device is proposed, including a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the executable instructions to implement the method described above.

[0017] Expanding the training dataset by introducing third-party auxiliary data relevant to the application scenario of the classification model allows the dataset to contain more information related to the classification task (especially multi-label classification). Cleaning and updating the labeled categories of the original data with third-party auxiliary information such as business processing logs from business implementers or feedback from business feedback providers can increase the number of labeled data samples and improve the labeling accuracy, thereby optimizing the classification performance of the trained classification model. Further supplementing the dataset with long-tail label category information can also effectively alleviate the imbalance of the original data and improve the classification performance of the classification model for objects with long-tail label categories. Attached Figure Description

[0018] The above and other features and advantages of this application will become more apparent from a detailed description of exemplary embodiments thereof with reference to the accompanying drawings.

[0019] Figure 1 Here is an exemplary logic diagram of a system flow for expanding a training dataset and training and building a classification model using the expanded dataset, according to one embodiment of this application;

[0020] Figure 2This is an exemplary flowchart of a method for expanding a training dataset according to an embodiment of this application;

[0021] Figure 3 For example Figure 2 An exemplary flowchart illustrating further details of the method for predicting and updating the categories of merged data in the training dataset is shown.

[0022] Figure 4 This is an exemplary flowchart of a method for training a classification model using an extended dataset according to an embodiment of this application;

[0023] Figure 5 This is an exemplary structural diagram of a classification model according to an embodiment of this application;

[0024] Figure 6 A schematic block diagram of an electronic device that classifies objects to be classified using a classification model trained with an expanded dataset, according to an embodiment of the present invention, for a method of expanding a training dataset. Detailed Implementation

[0025] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided to make the content of this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art. In the drawings, the dimensions of some elements may be exaggerated or modified for clarity. The same reference numerals in the drawings denote the same or similar structures, and therefore their detailed descriptions will be omitted.

[0026] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a full understanding of the solutions of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details described, or other methods, elements, etc., can be employed. In other instances, well-known structures, methods, or operations are not shown or described in detail to avoid obscuring various aspects of the inventive concept of this application.

[0027] In classification tasks, the objects to be classified may take different forms depending on the application scenario. For example, in many business scenarios across various industries such as the catering industry, business data needs to be classified for submission to the corresponding business personnel for processing; here, the business data is the object to be classified. This article uses business data from the catering industry as an example to introduce the solution of this application.

[0028] Business data can include data in various formats, such as, but not limited to, text, audio (speech, music, etc.), images (photos, composite images), video, and combinations of various formats. Text classification is a primary and common application scenario, and there is in-depth research and development focused on it. To facilitate the processing of business data by classification models, non-text data such as audio, images, and video can be converted into text data for classification using text classification models or algorithms. Before being input into the classification model, business data can include not only conversions between different data formats but also data preprocessing for structuring text data to meet the requirements of the text classification model.

[0029] In classification application scenarios, the business data to be classified typically includes information directly related to the business. For example, in a restaurant repair scenario, fault reporting information needs to be accurately classified to assign repair tasks to appropriate engineers. When reporting a fault, a fault description can be provided (e.g., by filling out a fault description form), and engineers will also record relevant work logs during the repair and handling of the reported fault. Another example is in a restaurant's customer service (customer support / complaints) scenario. The customer service system needs to automatically classify complaint information from customers (e.g., through customer complaints submitted at the restaurant, on a website, or in an application) to assign complaint cases to the corresponding customer service specialists. During the complaint handling process, customer service specialists can also record processing logs for later review or experience summarization. Yet another example is in a restaurant or advertiser's market recommendation or market research scenario. After the restaurant or advertiser categorizes and delivers research reports, product descriptions, or advertisements to potential customers, they will also obtain customer feedback on the research or advertisements. These work logs, processing logs, or feedback information include more business data that is related to the business scenario (or the application scenario of the task category) but different from the business scenario, such as process / flow data, experience data, subjective or objective understanding and / or feelings about the product / service.

[0030] When classifying the aforementioned business data (fault reporting data, customer complaint information, advertising information, and market research data, etc.), third-party auxiliary data can be used to expand the training dataset used to train the classification model. In this paper, third-party auxiliary data refers to data from sources different from the original data samples of the objects to be classified, but which are associated with the application scenario (business scenario) of the classification task / model. The term "third party" refers to parties in the business scenario other than the provider of the objects to be classified (e.g., business data) and the classifier of the objects (e.g., automated systems or servers used to classify business data). The term "auxiliary data" refers to "other" data from third parties that are associated with the business scenario but different from the data (business data) of the objects to be classified. Auxiliary data can provide more information that is beneficial to improving the performance of the classification model. The role of auxiliary data is not only to expand the number of labeled data samples but also to enhance the diversity and multi-dimensionality of the labeled data samples. For example, in a restaurant malfunction reporting scenario, the restaurant, as the provider of the malfunction reporting data, is the first party; the malfunction reporting data is the object to be classified; the automated machines (such as the devices implementing the classification model) that classify the malfunction reporting data are the second party; and the maintenance engineer is the third party. The work logs recorded by the maintenance engineer are auxiliary data. Similarly, in a restaurant customer service scenario, the customer, as the provider of complaint information, is the first party; the complaint information (such as converted text data) is the object to be classified; the server that classifies the complaint information is the second party; and the customer service specialist handling the customer complaint is the third party. The processing logs recorded by the customer service specialist are auxiliary data. Furthermore, in a market recommendation / research scenario, the product researcher / placer or advertiser (such as an advertiser or restaurant) is the first party; the research materials or advertisements are the object to be classified; the server that classifies the research / product description materials or advertisements is the second party; and the customers corresponding to the classified research / product materials or advertisements are the third party. The feedback information from these customers regarding the research / product / advertisement is auxiliary data. To distinguish it from auxiliary data, data containing no auxiliary data from third parties (such as business data) is called raw data. Raw data that has been labeled or categorized by manual or automatic means has (initial) categories or labels. In the following text, data categories and labels characterize the categorical attributes of data and can be used interchangeably. Generally speaking, data having a certain category / label or being classified into a certain category means that the data belongs to that category with a corresponding probability or has that label or is marked or labeled with that label with a corresponding probability.

[0031] The following section, in conjunction with the accompanying drawings, uses the example of expanding the dataset of fault reporting business data in a restaurant fault reporting scenario and constructing and training a system process for classifying fault reports to illustrate the solution of this application.

[0032] Figure 1 The system flow shown mainly includes a data expansion part (left side) that uses third-party auxiliary data to expand the business data to be classified, and a model building part (right side) that builds and trains a classification model based on the expanded business dataset.

[0033] The system process first merges the fault report content and the work log of the maintenance engineer responsible for repairing the fault in each fault report case as the raw data and third-party auxiliary data of the object to be classified (business data), respectively, to obtain merged data 101. The fault report case has been manually or otherwise labeled with the category or tags to which the fault belongs. If the maintenance engineer can handle the fault, then the maintenance engineer and the work log should correspond to the category to which the fault report content belongs, that is, the raw data and auxiliary data should correspond. Using the case as a data sample unit, the (initial) category of each merged data 101 should be the same as or related to the category of the original data of the business data (also known as the initial category of the original data).

[0034] In embodiments of this application, data merging operations can include simple data merging and data accumulation. Simple data merging can combine original data and auxiliary data into the same merged data. Simple merging can include treating the original data and auxiliary data as items in a data combination, such as one data item in a data pair (original data, auxiliary data), or it can include concatenating the original data and auxiliary data as components of the merged data, such as concatenating the text feature vectors of the original data in text form and the auxiliary data in text form. The initial category of the merged data 101 after simple merging is set to the (initial) category of the original data to reflect the correspondence between the original data and the auxiliary data. Other forms of simple merging can also be used, such as element-wise vector summation of text feature vectors. Data accumulation treats the original data and the auxiliary data corresponding to the original data as independent data samples to form two merged data 101, and then sets the initial category of both merged data to the category of the original data. In fact, the initial category of the merged data from the original data is still the (initial) category of the original data, and the initial category of the merged data from the auxiliary data corresponding to the original data is set to the (initial) category of the original data. Although the two merged data points 101 after data accumulation have the same category as the original data, they belong to different merged data sets. This is equivalent to processing the original data and auxiliary data separately, thereby expanding the number of data samples. In the merging operation, one or both methods can be used: simple merging and data accumulation. For example, a simple merging operation can be used on one part of the original data and its corresponding auxiliary data, while a data accumulation operation can be used on another part of the original data and its corresponding auxiliary data.

[0035] The generated merged data 101 is divided into a first merged data subset (also known as the training dataset) for training the classification model and a second merged data subset (also known as the validation dataset) for predicting the categories of fault reporting cases, as shown in box 110. Generally, the number of merged data points in the training dataset is greater than the number in the validation dataset. For example, the merged data 101 can be divided into 10 parts, with the training dataset, based on 8 parts (80%) of the merged data, being input (e.g., ...). Figure 1(As shown by arrow A) The original model of the classification model is trained (box 120) to determine the parameters of the text classification model. The original model of the classification model is trained to obtain the first classification model. The trained first classification model predicts the class of each merged data point in the validation dataset, which consists of the remaining 20% ​​of the merged data, to obtain one or more classes to which the merged data belongs and the probability corresponding to each class. Other rules can also be used to divide the training and validation datasets from the merged data 101, for example, using 70% (70%) of the merged data 101 as the training dataset and the remaining 3% (30%) as the validation dataset. Typically, the merged data in the training and validation datasets are not the same.

[0036] The merged data 101, which includes auxiliary data (such as work logs), can introduce more information than the original fault report data. Therefore, the trained first classification model can more accurately and comprehensively predict the category and probability of the merged data 101 in the validation dataset, thereby calibrating and updating the current category of the merged data 101.

[0037] The category predicted by the first classification model for merged data 101 is called the predicted category. If the predicted category of the merged data is the same as its current category, the current category of merged data 101 is considered accurate and does not need to be updated. Otherwise, it is necessary to determine whether the predicted category, which is different from the current category, is a more accurate category for merged data 101 (i.e., to determine whether the current category of the merged data is incorrect or inaccurate), and based on the determination result, to decide whether there is sufficient reason to update the current category of merged data 101 to the predicted category.

[0038] Those skilled in the art will understand that the predicted category of merged data 101 refers to the category (or label) with the highest corresponding probability value among one or more categories determined by the first classification model. The labeled initial category of the original data refers to the category that best reflects the main attributes of the original data, determined manually or by other means. Therefore, when the predicted category of merged data 101 differs from the current category, the probability corresponding to the predicted category is necessarily greater than the probability corresponding to the current category (e.g., the initial category) of merged data 101. The comparison between the corresponding probabilities of the predicted category and the current category can determine which category more accurately represents the category and attributes of merged data 101, i.e., which category is superior or more accurate.

[0039] A preset condition can be set for the probability difference between the probability of the predicted category of the merged data 101 in the validation dataset and the probability of the current category. If this preset condition is met, the current category is determined to be inaccurate, and the current category should be calibrated and updated to the predicted category. For example, the preset condition can be set such that an update from the current category to the predicted category occurs when the probability difference between the probability of the predicted category and the probability of the current category exceeds a probability difference threshold. The probability difference threshold can be determined based on the distribution of the probability differences between the probabilities of the predicted category and the current category of all merged data 101 in the validation dataset whose predicted categories differ from the current category. According to an embodiment of this application, the probability difference corresponding to a preset quantile in the above probability difference distribution can be selected as the probability difference threshold. The probability difference corresponding to a preset quantile refers to the minimum probability difference among fault report cases where the area enclosed by the probability difference distribution curve (e.g., calculated by integration) and the horizontal axis (which can be called the distribution area of ​​the probability difference distribution) exceeds the total area enclosed by the probability difference distribution curve and the horizontal axis of the coordinate system, corresponding to that preset quantile (e.g., percentage). In the probability difference distribution curve graph, the horizontal axis represents the fault report case number, and the vertical axis represents the probability difference value. For example, the preset quantile can be chosen as 95%, 90%, 85%, or a value between 99% and 1%, generally preferably closer to 100%. Quantiles reflect the confidence level of the probability difference; therefore, a higher quantile indicates a higher probability that the predicted category is more accurate than the current category.

[0040] In addition to using the predicted class of merged data 101 and the corresponding probability of the current class to determine whether to calibrate and update the class of merged data 101, the above judgment can also be made through the attributes of the feature vectors corresponding to the two.

[0041] Feature vectors corresponding to each category or label of merged data 101 can be generated. For example, in the case of formatted text data, the text feature vectors representing the category or label can be used as the feature vectors of that category. For the feature vectors corresponding to each category of synthetic data 101 in the validation dataset predicted by the first classification model, the average feature vector of all feature vectors is calculated as the centroid of the vector. The distance from the feature vector corresponding to each category to the centroid is the centroid distance from that category to the centroid. It can be understood that if a certain category of merged data 101 is the most accurate category that best represents the attributes of merged data 101, then the centroid distance corresponding to that category should be the minimum among the centroid distances corresponding to all categories, that is, that category should be closer to the average feature vector (centroid). The comparison between the centroid distance corresponding to the predicted category (first distance) and the centroid distance corresponding to the current category (second distance) can be used to determine which of the two more accurately represents the category and attributes of merged data 101, that is, which category is better.

[0042] A preset condition can be set for the distance difference between the first distance corresponding to the predicted category of the merged data 101 in the validation dataset and the second distance corresponding to the current category. If this preset condition is met, the current category is determined to be inaccurate, and the current category should be calibrated and updated to the predicted category. For example, the preset condition can be set such that if the distance difference between the first distance corresponding to the predicted category and the second distance corresponding to the current category exceeds a distance difference threshold, an update from the current category to the predicted category occurs. Similar to the probability difference threshold calculation method described above, it can be determined based on the distribution of the distance difference between the first distance corresponding to the predicted category and the second distance corresponding to the current category for all merged data 101 in the validation dataset whose predicted category differs from the current category. According to an embodiment of this application, the distance difference corresponding to a preset quantile in the aforementioned distance difference distribution can be selected as the distance difference threshold. The distance difference corresponding to a preset quantile refers to the minimum distance difference among fault report cases where the area enclosed by the distance difference distribution curve and the horizontal axis (which can be called the distribution area of ​​the distance difference distribution) exceeds the total area enclosed by the distance difference distribution curve and the horizontal axis of the coordinate system at that preset quantile (e.g., as a percentage). In the distance difference distribution curve graph, the horizontal axis represents the fault report case number, and the vertical axis represents the distance difference value. For example, the preset quantile can be chosen as 95%, 90%, 85%, or a value between 99% and 1%, generally preferably closer to 100%. Quantiles reflect the confidence level of the distance difference; therefore, a higher quantile indicates a higher probability that the predicted category is more accurate than the current category.

[0043] After updating the categories of all merged data 101 in the validation dataset using the above category calibration and update rules, the category-updated merged data in the validation dataset is placed into the training dataset to replace the same number of merged data in the training dataset as in the validation dataset. The merged data in the validation dataset is then replaced with the same number of merged data from the training dataset that were replaced, thus completing the data update for both the training and validation datasets. Figure 1 As shown by arrow B, the merged data replaced by the validation dataset did not undergo category updates during the current category prediction process of the first classification model. However, by adding the category-updated merged data to the training dataset, the accuracy of the labeled categories of the merged data in the training dataset was improved. For example, in the example of dividing 10 merged datasets 101 into a training dataset containing 8 merged datasets and a validation dataset containing 2 merged datasets, the 2 merged datasets out of the 8 datasets in the original training dataset that did not undergo category verification and updates are replaced with the 2 merged datasets in the validation dataset that have undergone category updates. The replaced 2 merged datasets serve as the new validation dataset for the next prediction by the first classification model. This update of the training and validation datasets is also called data cleaning of the labeled categories or labels of the merged data 101, in which at least one or more attributes or data components of the merged data 101 are corrected.

[0044] Next, the first classification model is retrained using the new training dataset to update and iterate the classification model, as shown in box 130. The retrained and updated first classification model predicts the class of the merged data 101 in the new validation dataset and updates the classes of all merged data 101 in the validation dataset according to the class calibration and update rules described above, replacing the merged data in the training dataset to obtain an updated training dataset, as shown in box 140. The updated training dataset is used again as the training dataset to train the first classification model, as shown by arrow D. The data cleaning and class update cycle shown by arrows A to D is repeated until the iteratively updated first classification model's class prediction performance for the merged data in the updated validation dataset meets the requirements.

[0045] The predictive performance of a classification model can be represented by its prediction error rate (RER). The RER represents the ratio between the number of merged data points that need to be validated and updated to predict the current class of merged data 101 in the validation dataset, and the total number of merged data points 101 in the validation dataset. A lower RER indicates more accurate labeling or updated classes of merged data 101 in the training dataset used to train the classification model, resulting in a better-performing classification model. Consequently, the labeling or updated predicted classes of the merged data in the validation dataset used to validate the model's predictive performance are also more accurate. A preset condition for the RER can be set so that the classification model's RER is below a preset RER threshold. The RER threshold can be set to a value greater than 0% and less than 50%, such as 10% or 5%.

[0046] The data expansion loop terminates when the prediction error rate of the first classification model falls below a preset prediction error rate threshold. A first expanded dataset 102 can be generated based on the latest training and validation datasets for use in... Figure 1 The model building section on the right constructs the classification model. In the following text, to distinguish it from the first classification model used in the data expansion section, the classification model generated in the model building section based on either the first expanded dataset 102 or the second expanded dataset 104 (see below) is referred to as the second classification model. The second classification model can be generated by training the original classification model based on the expanded first and / or second expanded datasets, or by training the first classification model mentioned above based on the expanded first and / or second expanded datasets. Since the categories or labels labeled in the merged data 101 in the latest training and validation datasets have high accuracy, the first expanded dataset 102 can be generated using all or a portion of the merged data 101 in the latest training and / or validation datasets, after data cleaning, category calibration, and updates.

[0047] According to embodiments of this application, a second classification model for classifying business data to be classified may include a feature extraction sub-model for extracting text features from input structured text data to obtain text feature vectors, and a prediction sub-model for predicting the type of business data based on the text feature vectors.

[0048] The feature extraction sub-model can be implemented using model structures or algorithms such as deep learning models or neural network models. For example, the feature extraction sub-model can be a natural language processing (NLP) model such as the BERT model or the Roberta model. According to embodiments of this application, the feature extraction sub-model can be a pre-trained model trained with massive amounts of training data, which has good text feature extraction performance. The feature extraction sub-model can be retrained based on merged data from a first extended dataset 102, including third-party auxiliary data and data expansion (i.e., fine-tuning the model parameters, as shown in box 150) to better suit the text feature extraction task of the specific application scenario of business data classification in this application.

[0049] The first extended dataset 102 can be used to train the model parameters of the prediction sub-model of the second classification model. The fine-tuned feature extraction sub-model and the prediction sub-model trained on the first extended dataset 102 based on the fine-tuned feature extraction sub-model constitute the second classification model used to complete the classification task, such as... Figure 1 The text classification model 105 is shown in the figure. When implementing the text classification model 105 using a neural network architecture, the prediction sub-model can also be implemented as a classification layer that is used to predict the category of text feature vectors. The classification model can include multiple layers of feature extraction sub-model and prediction sub-model (e.g., classification layer) from the input layer (bottom layer) to the output layer (top layer).

[0050] When the business data to be classified contains long-tail labels or category data, the merged data in the first extended dataset 102, after category updates, can be further expanded to generate merged data with long-tail labels (long-tail label merged data) 103. This avoids the problem of merged data belonging to long-tail label categories being incorrectly predicted as ordinary categories that are not long-tail label categories due to data imbalance. The process of generating long-tail label merged data 103 is equivalent to further expanding the number of categories in the merged data after data cleaning, category calibration, and updates based on the first extended dataset 102 and long-tail label information. Long-tail label merged data 103 can be generated by training a text generation model for generating long-tail label merged data 103 using the first extended dataset 102 and long-tail label information associated with the business data application scenario, such as... Figure 1 As shown in box 160. The text generation model supplements the merged data with insufficient labeled data and long-tailed labels or categories, enabling the text classification model 105 to fully learn the features of each category. The text generation model can be implemented using a deep learning model or a neural network model.

[0051] Based on the generated long-tailed label merged data 103, the long-tailed label merged data 103 can be added to the first extended dataset 102, and together with the merged data, a second extended dataset 104 can be generated. The operation of generating new merged data based on the long-tailed label merged data 103 and the merged data in the first extended dataset 102 can adopt a similar operation to the merging operation described above for generating merged data 101, such as simple data merging or data accumulation. The second extended dataset 104 can be used again to train the text classification model 105 (fine-tuning the feature extraction sub-model 150 and training the prediction sub-model based on the fine-tuned feature extraction sub-model) or only to train the prediction sub-model (classification layer) in the text classification model 105 (where the feature extraction sub-model 150 has been fine-tuned using the first extended dataset 102). In the model construction process that incorporates the long-tailed label merged data 103, the final training dataset (i.e., the second extended dataset 104) includes richer information related to the classification application scenario of business data compared to the first extended dataset 102 generated from the original data and auxiliary data, thereby further improving the category prediction performance of the text classification model 105.

[0052] Figure 2 An exemplary flow of a method for expanding a dataset according to embodiments of this application is shown. Wherein with Figure 1 The parts of the system logic that are the same as or similar to those described in the previous section will not be described in detail.

[0053] The method first generates merged data in step S210 based on the original data of the business data to be classified, which only includes business scenarios (such as text data of fault repair cases), and auxiliary data from a third party with a different source than the original data. Then, in step S220, the categories of the merged data are updated using a first classification model to generate a first extended dataset of merged data that has been cleaned, type-calibrated and updated.

[0054] In step S210, different data merging operations can be used to obtain merged data. For example, merged data can be generated by sub-step S211, which simply merges each original data and its corresponding auxiliary data into the same merged data. Alternatively, merged data can be generated by sub-step S212, which accumulates each original data and its corresponding auxiliary data to obtain two independent and different merged data. In the simple merging method, the initial category of the same merged data is set to the category of the original data, while in the accumulation method, the initial category of each merged data in the two merged data is set to the category of the original data.

[0055] Step S220 further includes: First, in sub-step S221, the merged data is divided into a first merged data subset, which includes different merged data, as a training dataset, and a second merged data subset, which is used as a validation dataset. Then, in sub-step S222, the original model of the classification model or the prediction sub-model in the original model of the classification model is trained using the training dataset to obtain a trained first classification model. In step S223, the trained first classification model is used to predict the predicted class of the merged data in the validation dataset, and the current class of the merged data is updated in accordance with the class calibration and update rules. Then, a first extended dataset is generated based on the first merged data subset and the updated second merged data subset. The generation of the first extended dataset can be achieved by reusing merged data from the second merged data subset to replace merged data from the first merged data subset to update the first and second merged data subsets, and by generating the first extended dataset based on the updated first and updated second merged data subsets. Specifically, the process may include repeatedly training a first classification model using the updated first merged data subset to update the first classification model, and repeatedly using the predicted categories of the merged data in the second merged data subset of the updated first classification model to update the categories of the merged data in the second merged data subset, until the prediction error rate of the first classification model for the categories of the merged data in the updated second merged data subset meets a preset error rate condition. As shown in the figure, in step S224, the merged data in the training dataset is replaced with the merged data in the validation dataset after the category update, thereby updating both datasets. If the category prediction error rate shown in sub-step S225 does not meet the preset condition (the judgment result is "No"), then step S220 repeats sub-steps S222 to S224 until the category prediction error rate of the first classification model shown in sub-step S225 meets the preset condition (the judgment result is "Yes"), then proceeds to sub-step S226, where a first extended dataset is generated based on the merged data in the latest training dataset and / or validation dataset.

[0056] Figure 3The details of sub-step S223 are further shown below. The category calibration and update rule first determines in sub-step S223a whether the predicted category of the merged data predicted by the first classification model is different from the current category of the merged data. If the predicted category in sub-step S223a is different from the current category (result: "Yes"), it indicates that the predicted category predicted by the classification model may be more accurate than the current category. The method then proceeds to sub-steps S223b and S223c, using different methods to determine which category is better, the predicted category or the current category, and whether to update the category of the merged data. If the predicted category in sub-step S223a is the same as the current category (result: "No"), it indicates that under the current training dataset and the first classification model, the predicted category and the labeled category have the same accuracy, or the labeled current category is accurate enough. The method then directly proceeds to sub-step S223e, continuing to execute sub-step S223a on the next merged data in the validation dataset, until all merged data in the validation dataset has completed the category update determination.

[0057] In the judgment method of sub-step S223b, the decision on whether to update the category of the merged data is based on probability. Specifically, if the probability difference between the probability corresponding to the predicted category and the probability corresponding to the current category meets the preset probability difference condition (judgment result is "yes"), the current category is considered inaccurate (at least compared to the predicted category), and the current category of the merged data should be updated to the predicted category (sub-step S223d). Otherwise, it is considered that the current predicted category does not have better accuracy than the current category, and the method proceeds to sub-step S223e. In the judgment method of sub-step S223c, the decision on whether to update the category of the merged data is based on vector distance. Specifically, if the distance difference between the distance from the feature vector corresponding to the predicted category to the average feature vector (centroid) and the distance from the feature vector corresponding to the current category to the centroid meets the preset distance difference condition (judgment result is "yes"), the current category is considered inaccurate (at least compared to the predicted category), and the current category of the merged data should be updated to the predicted category (sub-step S223d). Otherwise, it is considered that the current predicted category does not have better accuracy than the current category, and the method proceeds to sub-step S223e.

[0058] According to an embodiment of this application, long-tail label information can also be introduced in step S230 to further expand the first extended dataset. Step S230 includes generating merged data with long-tail labels based on the merged data and long-tail label information of the first extended dataset (sub-step S231), and adding the generated long-tail label merged data to the first extended dataset to further expand the first extended dataset to generate a second extended dataset (sub-step S232).

[0059] Figure 4An exemplary flow illustrating a method for constructing and training a classification model based on an extended dataset (first or second extended dataset) generated by the method for extended (training) dataset described above, according to embodiments of this application.

[0060] Combination Figure 5 The exemplary structure of the classification model shown is such that classification model 400 may include feature extraction sub-model 410 and prediction sub-model 430. Figure 4 The method first fine-tunes the feature extraction sub-model implemented by the pre-trained model using the first extended dataset 102 in step S310, and then trains the prediction sub-model using the first extended dataset 102 on the basis of the fine-tuned feature extraction sub-model 410 in step S320, so as to finally obtain the second classification model 400.

[0061] According to embodiments of this application, a text generation model 420 can also be used to further expand the extended dataset 102 by incorporating long-tail label information. The text generation model 420 can be part of the second classification model 400 or independent of the second classification model 400. Figure 5 The dashed box 420 in the figure illustrates the case where the text generation model 420 is independent of the second classification model 400. If the text generation model 420 is not used, step S320 primarily trains the prediction sub-model based on the first extended dataset 102, as shown in sub-step S321. If the text generation model 420 exists, step S320 may further include sub-steps S322a to S322d, which are parallel to sub-step S321. Specifically, in sub-step S322a, step S320 first trains the text generation model 420 based on the data-cleaned, category-calibrated, and updated merged data of the first extended dataset 102, and then in sub-step S322b, the text generation model 420 is used to generate long-tailed labeled merged data. Step S322c adds the long-tailed label merged data to the first extended dataset to generate a further extended second extended dataset. Then, in sub-step S322d, the method fine-tunes the feature extraction sub-model 410 of the second classification model 400 based on the second extended dataset and trains the prediction sub-model 430 of the second classification model 400 based on the fine-tuned feature extraction sub-model 410. Alternatively, the method trains the prediction sub-model 430 of the second classification model 400 using the second extended dataset based on the first extended data and the fine-tuned feature extraction sub-model 410.

[0062] After training as described above, a higher-performing classification model is obtained. When predicting the classification of business data to be classified, more information from the original business data sources and dimensions, as well as long-tail label information, can be considered to improve the accuracy of classification and / or the adaptability of classification application scenarios.

[0063] As described above, the expanded training dataset and the scheme for training a classification model based on the expanded training dataset expand the training dataset by introducing third-party auxiliary data related to the application scenario of the classification model. The training dataset can contain more information relevant to the classification task (especially multi-label classification). By cleaning and updating the labeled categories of the original data with third-party auxiliary information such as business processing logs from business implementers or feedback information from business feedback providers, the number of labeled data samples can be increased and the labeling accuracy can be improved, thereby optimizing the classification performance of the trained classification model. Further supplementing the expanded dataset with long-tail label category information can also effectively alleviate the imbalance of the original data and improve the classification performance of the classification model for objects with long-tail label categories.

[0064] It should be noted that although several modules or units for expanding datasets and building and training classification models based on expanded datasets are described in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units. Components shown as modules or units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the solution in this application according to actual needs. Those skilled in the art can understand and implement this without inventive effort.

[0065] In exemplary embodiments of this application, a computer-readable storage medium is also provided, on which a computer program is stored. The program includes executable instructions that, when executed by, for example, a processor, can implement the steps of the method for expanding a dataset and constructing and training a classification model based on the expanded dataset as described in any of the above embodiments. In some possible implementations, various aspects of this application can also be implemented as a program product comprising program code that, when run on a terminal device, causes the terminal device to perform the steps described in this specification for expanding a dataset and constructing and training a classification model based on the expanded dataset, according to various exemplary embodiments of this application.

[0066] The program product for implementing the above-described method according to embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this application is not limited thereto. In this document, a readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0067] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0068] The computer-readable storage medium may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0069] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0070] In an exemplary embodiment of this application, an electronic device is also provided, which may include a processor and a memory for storing executable instructions of the processor. The processor is configured to execute the steps of the method for expanding a dataset and constructing and training a classification model based on the expanded dataset in any of the above embodiments by executing the executable instructions.

[0071] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0072] The following reference Figure 6 To describe an electronic device 600 according to this embodiment of the present application. Figure 6 The electronic device 600 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0073] like Figure 6 As shown, the electronic device 600 is presented in the form of a general-purpose computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different system components (including storage unit 620 and processing unit 610), a display unit 640, etc.

[0074] The storage unit stores program code that can be executed by the processing unit 610, causing the processing unit 610 to perform the steps of various exemplary embodiments according to this application described in the methods for expanding datasets and constructing and training classification models based on expanded datasets. For example, the processing unit 610 can perform actions such as... Figures 2 to 4 The steps are shown in the figure.

[0075] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache storage unit 6202, and may further include a read-only memory unit (ROM) 6203.

[0076] The storage unit 620 may also include a program / utility 6204 having a set (at least one) program module 6205, such program module 6205 including but not limited to: an operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0077] Bus 630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0078] Electronic device 600 can also communicate with one or more external devices 500 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 600, and / or with any device that enables electronic device 600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 650. Furthermore, electronic device 600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 660. Network adapter 660 can communicate with other modules of electronic device 600 via bus 630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0079] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product. This software product can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, and includes several instructions to cause a computing device (such as a personal computer, server, or network device, etc.) to execute the method according to the embodiments of this application for expanding a dataset and constructing and training a classification model based on the expanded dataset.

[0080] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the appended claims.

Claims

1. A method for expanding a training dataset, the training dataset comprising multiple original data sets, the method comprising: Generate merged data based on the original data and auxiliary data, wherein the auxiliary data is associated with the application scenario of the classification model and comes from a different source than the original data. The generation of merged data based on the original data and auxiliary data includes: The original data and the auxiliary data are simply merged to obtain merged data, and the category of the merged data is set to the category of the original data; and / or The original data and the auxiliary data are summed to obtain the merged data after data summation. The categories of the merged data after data summation are all set to the categories of the original data. The classification model is used to update the categories of the merged data to generate a first expanded dataset for training the classification model, including: The merged data is divided into a first merged data subset and a second merged data subset, which have different merged data. The original model of the classification model is trained using the first merged subset of data to obtain the first classification model; The categories of the merged data in the second merged data subset are updated using the predicted categories of the merged data in the second merged data subset determined by the first classification model. The merged data in the second merged data subset is reused to replace at least a portion of the merged data in the first merged data subset to update the first merged data subset; The merged data in the second merged data subset is updated by repeatedly using the replaced at least part of the merged data in the first merged data subset; The first extended dataset is generated based on the updated first merged data subset and the updated second merged data subset, wherein the first extended dataset includes merged data with updated categories.

2. The method according to claim 1, characterized in that, Generating the first expanded dataset based on the updated first merged data subset and the updated second merged data subset further includes: The first classification model is trained again using the updated first merged data subset to update the first classification model; The predicted categories of the merged data in the second merged data subset, determined by the updated first classification model, are repeatedly used to update the categories of the merged data in the second merged data subset until the prediction error rate of the first classification model for the categories of the merged data in the updated second merged data subset meets the preset error rate condition.

3. The method according to claim 1, characterized in that, Updating the category of the merged data in the second merged data subset using the predicted category of the merged data in the second merged data subset determined by the first classification model further includes: If the predicted category of the merged data in the second merged data subset is different from the current category of the merged data, the current category of the merged data is updated to the predicted category.

4. The method according to claim 3, characterized in that, Updating the current category of the merged data to the predicted category includes: If the probability difference between the probability corresponding to the predicted category of the merged data in the second merged data subset and the probability corresponding to the current category of the merged data satisfies a preset probability difference condition, the current category of the merged data is updated to the predicted category.

5. The method according to claim 4, characterized in that, The probability difference preset condition includes the probability difference between the probability corresponding to the predicted category and the probability corresponding to the current category in the merged data exceeding the probability difference threshold.

6. The method according to claim 5, characterized in that, The probability difference threshold is determined based on the probability difference corresponding to the area of ​​the distribution of a preset quantile in the distribution of the probability difference.

7. The method according to claim 3, characterized in that, Updating the current category of the merged data to the predicted category includes: The average vector of the feature vectors is determined based on the feature vectors corresponding to the categories of the merged data; Calculate a first distance between the feature vector corresponding to the predicted category of the merged data in the second merged data subset and the average vector, and a second distance between the feature vector corresponding to the current category of the merged data and the average vector; If the distance difference between the first distance and the second distance meets the preset distance difference condition, the current category of the merged data is updated to the predicted category.

8. The method according to claim 7, characterized in that, The preset condition for the distance difference includes the distance difference exceeding a distance difference threshold.

9. The method according to claim 8, characterized in that, The distance difference threshold is determined based on the distance difference corresponding to the distribution area of ​​a preset quantile in the distribution of the distance difference.

10. The method according to claim 1, characterized in that, The first merged data subset contains more merged data than the second merged data subset.

11. The method according to claim 1, characterized in that, The generated merged data based on the original data and auxiliary data includes: Merge the original data and the corresponding auxiliary data into a single merged data, and set the initial category of the merged data to the category of the original data; and / or The original data and the auxiliary data corresponding to the original data are treated as different merged data, and the initial category of the different merged data is set as the category of the original data.

12. The method according to claim 1, characterized in that, The method further includes: The first extended dataset is expanded based on long-tail tag information associated with the application scenario to generate a second extended dataset.

13. The method according to claim 12, characterized in that, Expanding the first expanded dataset to generate the second expanded dataset includes: Based on the merged data in the first extended dataset and the long-tail label information, generate merged data with long-tail labels; The merged data with long-tail labels is added to the first extended dataset to generate the second extended dataset.

14. The method according to claim 13, characterized in that, Generating merged data with long-tail labels based on the merged data in the first extended dataset and the long-tail label information further includes: The text generation model for generating data with long-tail labels is trained using the first expanded dataset and the long-tail label information; and The text generation model is used to generate merged data with long-tail labels based on the first extended dataset.

15. The method according to any one of claims 1 to 14, characterized in that, The method includes: The original model or the first classification model is trained using the first extended dataset and / or the second extended dataset according to any one of claims 12 to 14 to generate a second classification model.

16. The method according to claim 15, wherein the second classification model includes a feature extraction sub-model for extracting features of the object to be classified, characterized in that, The method further includes adjusting the feature extraction sub-model using the first extended dataset and / or the second extended dataset.

17. The method according to claim 15, wherein the second classification model includes a prediction sub-model for predicting the category of the object to be classified, characterized in that, The method further includes training the prediction sub-model using the first extended dataset and / or the second extended dataset.

18. The method according to any one of claims 1 to 14, characterized in that, The source of the raw data is the business requester in the application scenario, and the source of the auxiliary data is the business executor or business feedback provider in the application scenario.

19. The method according to claim 18, characterized in that, The classification model includes a text classification model.

20. The method according to claim 18, characterized in that, The classification model includes a neural network model or a deep learning model.

21. A method for predicting the category of an object to be classified, comprising: Obtain the object to be classified; The second classification model trained using the method according to any one of claims 15 to 17 predicts the category of the object to be classified.

22. A computer-readable storage medium having a computer program stored thereon, the computer program including executable instructions that, when executed by a processor, implement the method according to any one of claims 1 to 21.

23. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the executable instructions to implement the method according to any one of claims 1 to 21.

Images