A method of processing training data, a storage medium, and an electronic device

Abstract

Embodiments of the present application provide a method for processing training data, a storage medium and an electronic device, the method comprising: obtaining labels of each category from a data set to obtain a category label set; selecting one multi-label data from the data set for each label in the category label set, taking the category label set and all selected multi-label data as a basic validation set; obtaining a validation set according to a size relationship between a total amount of data included in the basic validation set and a target data amount required by a validation set; and obtaining data of a training set according to data in the data set except the validation set. The standardized method of embodiments of the present application effectively solves the problem of unreasonable labels in multi-label data set division, can make full use of the entire data set, and thus utilizes existing data to train a relatively robust system model.

Description

Technical Field

[0001] This application relates to the field of natural language data processing, and more specifically, embodiments of this application relate to a method for processing training data, a storage medium, and an electronic device. Background Technology

[0002] In practical deployments, natural language processing systems often encounter multi-labeled data, meaning that each input data point corresponds to multiple labels. For example, news on a portal website can be categorized into different sections (such as politics, sports, society, technology, finance, etc.). If an article describes both political and financial content, then that article might be labeled with both "politics" and "finance."

[0003] Multi-task natural language processing (NLP) systems require standardized multi-label datasets during training. NLP systems need to extract different proportions of data as standardized training and validation sets during training; imbalanced dataset partitioning can negatively impact the system's training and evaluation processes. Standardized dataset partitioning can prevent the system from being overly sensitive or undersensitive to data with specific labels, thus avoiding errors in system decision-making due to data issues.

[0004] Standardization of typical single-label datasets usually involves randomly dividing the training and validation sets according to a set ratio. However, in practical applications, it is difficult to randomly divide multi-label datasets.

[0005] Therefore, the inventors of this application have found that how to reasonably divide a multi-label dataset into training and validation sets has become a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The purpose of this application is to provide a method, storage medium, and electronic device for processing training data. In standardizing multi-label datasets (i.e., dividing them into training and validation sets), this application employs a model that first divides the validation set and then the training set. Furthermore, when dividing the validation set, it ensures that the set has as many label categories as possible, the category imbalance is as small as possible, and the label quantity imbalance is as small as possible, thereby guaranteeing the rationality of the entire dataset division. This standardization method effectively solves the problem of unreasonable label division in multi-label datasets, and can fully utilize the entire dataset, thereby training a relatively robust system model using existing data.

[0007] In a first aspect, embodiments of this application provide a method for processing training data, the method comprising: obtaining labels for each category from a dataset to obtain a category label set; selecting a multi-label data set from the dataset for each label in the category label set, and using the category label set and all selected multi-label data sets as a basic validation set; obtaining a validation set based on the size relationship between the total amount of data included in the basic validation set and the target amount of data required for the validation set; and obtaining a training set based on the data in the dataset other than the validation set.

[0008] Some embodiments of this application can ensure that the validation set contains various types of labels as much as possible by extracting various types of labels and selecting a multi-label data for each type of label, thereby improving the label coverage of the obtained validation set and thus improving the technical effect of training the model based on these data.

[0009] In some embodiments, obtaining the validation set based on the size relationship between the total amount of data included in the basic validation set and the target amount of data required for the validation set includes: if it is confirmed that the total amount of data included in the basic validation set is not equal to the target amount of data required for the validation set, then obtaining the validation set based on the dataset to be processed and the validation set to be processed, wherein the initial set of the dataset to be processed is the dataset or a dataset determined by the basic validation set, and the initial set of the validation set to be processed is an empty set or the basic validation set.

[0010] In some embodiments of this application, when it is confirmed that the amount of data contained in the basic validation set does not meet the target amount of data required for the validation set, a validation set is obtained based on the dataset to be processed and the validation set to be processed, thereby improving the versatility of the technical solution.

[0011] In some embodiments, if the total amount of data included in the basic validation set is less than the target amount of data required by the validation set, then the initial set of the dataset to be processed is the data included in the dataset excluding the basic validation set, and the initial set of the validation set to be processed is the basic validation set; if the total amount of data included in the basic validation set is greater than the target amount of data required by the validation set, then the initial set of the dataset to be processed is the basic validation set, and the initial set of the validation set to be processed is empty.

[0012] Some embodiments of this application provide various methods for obtaining a validation set from a dataset to be processed and a validation set to be processed.

[0013] In some embodiments, obtaining the validation set based on the dataset to be processed and the validation set to be processed includes: selecting K data points from the dataset to be processed and removing the K data points from the dataset to be processed; adding the K data points to the validation set to be processed; if it is determined that the amount of data in the validation set to be processed is less than the target amount of data required by the validation set, repeating the above process until the amount of data in the validation set to be processed is equal to the target amount of data required by the validation set, and then using the validation set to be processed as the validation set.

[0014] Some embodiments of this application provide a process for selecting data from a dataset to be processed to form a validation set. In each selection, a portion of the data is selected according to the above selection principles to improve the final validation set by having as many label categories as possible, as little class label imbalance as possible, and as little label quantity imbalance as possible, thereby improving the quality of the trained model.

[0015] In some embodiments, selecting K data points from the dataset to be processed includes: selecting the K data points from the dataset to be processed based on label coverage, class data imbalance, and label quantity imbalance. The label coverage is used to quantify the total number of label categories in the candidate data. The class data imbalance is used to quantify the difference between the actual data volume corresponding to each label category in the candidate data and the target data volume of the corresponding label category. The label quantity imbalance is used to quantify the difference between a first ratio and a second ratio. The first ratio is the ratio of the total number of labels corresponding to the candidate data to the total data volume included in the candidate data. The second ratio is the ratio of the total number of labels corresponding to the dataset to the total number of data points in the dataset.

[0016] Some embodiments of this application provide functions related to quantifying selection principles. These functions can better quantify and evaluate the K data points of each candidate group, and then select the K data points used to form the validation set.

[0017] In some embodiments, the class data imbalance is the sum of the differences between the actual data amount of each class label corresponding to any K selected data points and the target data amount of the corresponding class label after standardization, wherein the standardization belongs to any axisymmetric function with a positive result.

[0018] Some embodiments of this application provide design principles for the class data imbalance function, which improves the versatility of the technical solution and enhances the accuracy and objectivity of the obtained class data imbalance values.

[0019] In some embodiments, the formula for calculating the class data imbalance is:

[0020]

[0021] Where M represents the total number of categories of labels corresponding to any K data points, and D... i DT represents the actual amount of data contained in the i-th type of label. i The target data volume corresponding to the i-th label is determined based on the proportion of the validation set in the dataset.

[0022] Some embodiments of this application provide a formula for calculating the imbalance of category data, thereby improving the accuracy and objectivity of the obtained value.

[0023] In some embodiments, the label imbalance is the sum of the differences between the number of all labels corresponding to any K selected data points and the number of the second target labels after standardization. The standardization can use any axisymmetric function with a positive result. The number of the second target labels is the product of the ratio of the data volume of the validation set to the data volume of the dataset and the total number of labels in the dataset.

[0024] Some embodiments of this application provide design principles for a function of label quantity imbalance, which improves the versatility of the technical solution and enhances the accuracy and objectivity of the obtained label quantity imbalance value.

[0025] In some embodiments, the formula for the label quantity imbalance is as follows:

[0026] tagnum_unbalance(KG) = (L - LT) 2

[0027] Where L represents the total number of all labels corresponding to any K data points, and LT represents the value of the second target label count.

[0028] Some embodiments of this application provide a formula for calculating the label quantity imbalance, thereby improving the accuracy and objectivity of the obtained value.

[0029] In some embodiments, selecting the K data points from the dataset to be processed based on label coverage, class data imbalance, and label number imbalance includes: using the dataset to be processed as an initial candidate set; obtaining the top n combinations with the highest label coverage from the initial candidate set using an optimal solution search algorithm to obtain a first candidate set, wherein one combination includes K candidate data points; calculating the class data imbalance of all combinations in the first candidate set and selecting the top m combinations in order to obtain a second candidate set, wherein one combination includes K candidate data points; calculating the label number imbalance of all combinations in the second candidate set and selecting the top k combinations in order to obtain a third candidate set, wherein one combination includes K candidate data points; and selecting the first data point from the third candidate set as the K data points.

[0030] Some embodiments of this application select K data points from the dataset to be processed based on label coverage, class data imbalance, and label number imbalance, so that the class data imbalance of the selected K data points is as small as possible and the label number imbalance of the selected K data points is also as small as possible, thereby improving the quality of the obtained validation set.

[0031] In some embodiments, selecting K data points from the dataset to be processed includes: selecting the K data points from the dataset to be processed according to a target normalization function, wherein the target normalization function is a function related to the label coverage, the class data imbalance, and the label quantity imbalance.

[0032] Some embodiments of this application select K data points by defining a uniform function.

[0033] In some embodiments, the target normalization function is obtained by weighting a normalized label coverage function, a normalized category data imbalance function, and a normalized label quantity imbalance function. The normalized label coverage function is obtained by normalizing the label coverage, the normalized category data imbalance function is obtained by normalizing the category data imbalance, and the normalized label quantity imbalance function is obtained by normalizing the label quantity imbalance.

[0034] Some embodiments of this application provide a design principle for a target normalization function that improves the quality of a validation set consisting of K data points obtained from the function.

[0035] Secondly, some embodiments of this application provide a method for training a natural language processing system, the method comprising: training the natural language processing system according to a training set and a validation set obtained by the method described in any embodiment of the first aspect.

[0036] Thirdly, some embodiments of this application provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, can implement the method described in any embodiment of the first aspect.

[0037] Fifthly, some embodiments of this application provide an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, can implement the method as described in any embodiment of the first aspect. Attached Figure Description

[0038] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0039] Figure 1 A schematic diagram of multi-tag data provided in an embodiment of this application;

[0040] Figure 2 This is a schematic diagram illustrating the training process of a natural language processing system based on a standardized multi-label dataset, provided as an embodiment of this application.

[0041] Figure 3 One of the flowcharts for a method of processing training data provided in the embodiments of this application;

[0042] Figure 4 A second flowchart illustrating the method for processing training data provided in this application embodiment;

[0043] Figure 5 The third flowchart of the method for processing training data provided in the embodiments of this application;

[0044] Figure 6 A flowchart of the method for processing training data provided in the embodiments of this application is shown in Part 4.

[0045] Figure 7 The fifth flowchart of the method for processing training data provided in the embodiments of this application;

[0046] Figure 8 A flowchart of a method for processing training data provided in an embodiment of this application;

[0047] Figure 9 A block diagram of the apparatus for processing training data provided in the embodiments of this application;

[0048] Figure 10This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0049] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0050] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0051] The inventors of this application discovered in their research that the standardization of general single-label datasets usually involves directly and randomly dividing the training and validation sets according to a certain ratio. However, in practical applications, if the method of randomly dividing multi-label datasets is also used to obtain validation and training set data, there are at least three disadvantages: 1) The label ratio of the divided dataset is significantly different from the original ratio; 2) The divided dataset cannot contain all rare labels, i.e., there is a problem of insufficient label categories; 3) Randomly dividing the data is relatively inefficient and cannot make full use of the data, i.e., it is not possible to train a robust system model using the existing data.

[0052] Please refer to Figure 1 Multi-tag data in some embodiments of this application, such as Figure 1 As shown, this refers to a single data point (e.g., Figure 1 An article with 101 data points has multiple tags (e.g., Figure 1 This data, for an article, has two tag categories: politics and news. Figure 1 One of the articles, number 101, belongs to multi-label data.

[0053] As described in the background section, in order to train a natural language processing system (e.g., a multi-task natural language processing system) to perform a specific task, it is necessary to... Figure 2A standardized multi-label dataset 130 is input into the natural language processing system 120 to train the system. Therefore, the quality of the standardized multi-label dataset 130 determines the performance of the trained natural language processing system 120. The standardized multi-label dataset 130 includes a validation set and a training set. It should be noted that the training set and validation set obtained by the method provided in this application contain all labels for different tasks, and the label ratio is balanced. It is easy to understand that an imbalanced dataset partition will affect the system's training or evaluation process. For example, incomplete labels in the training set will lead to the system's insensitivity to missing labels, and incomplete labels in the validation set will prevent the validation process from correctly evaluating the system's actual performance, thus affecting the training effect.

[0054] The following is combined with Figure 3 This application provides exemplary methods for processing training data according to some embodiments, through which training data can be obtained. Figure 2 A standardized multi-labeled dataset 130.

[0055] like Figure 3 As shown in the embodiment of this application, a method for processing training data is provided, the method comprising:

[0056] S101: Obtain the labels for each category from the dataset to obtain a set of category labels.

[0057] For example, if all the data in the dataset corresponds to four labels: politics, sports, society, and technology, then executing S101 will treat these four labels as a category label set.

[0058] S102, select one multi-label data from the dataset for each label in the category label set, and use the category label set and all selected multi-label data as the basic validation set.

[0059] For example, if all the data in the dataset corresponds to four categories of labels: politics, sports, society, and technology, then executing S101 means taking these four categories of labels as a set of category labels, and executing S102 means selecting a multi-label data point from the dataset for each of these four categories of labels, and using these four categories of labels and the multi-label data points selected for each category of labels as the basic validation set.

[0060] For example, such as Figure 4As shown, the process of obtaining the basic validation set in this application embodiment includes, for example,: S110, firstly, randomly shuffling all the data in the dataset to obtain an initial dataset; S120, selecting labels for each category from the initial dataset, and selecting a multi-label data for each category label to form an initial validation set; S130, deduplicating the initial validation set to obtain a basic validation set, and the data size of the basic validation set is BN.

[0061] S103, the validation set is obtained based on the relationship between the total amount of data included in the basic validation set and the target amount of data required for the validation set.

[0062] For example, in some embodiments of this application, S103 includes:

[0063] If it is determined that the total amount of data included in the basic validation set is equal to the target amount of data required by the validation set, then the basic validation set is used as the validation set.

[0064] If it is confirmed that the total amount of data included in the basic validation set is not equal to the target amount of data required by the validation set, then the validation set is obtained based on the dataset to be processed and the validation set to be processed. The initial set of the dataset to be processed is the dataset or a dataset determined by the basic validation set, and the initial set of the validation set to be processed is an empty set or the basic validation set.

[0065] In other words, such as Figure 4 As shown, S103 includes, for example, S140, dividing the dataset into training and validation sets, determining the ratio (rate) of the training and validation sets, and obtaining the target data size TN for the validation set; S150, determining whether TN equals BN, if so, then executing S160 to use the basic validation set as the validation set and using the data in the dataset other than the basic validation set as the training set data; otherwise, executing S170 to obtain the validation set based on the dataset to be processed and executing S180 to use the data in the dataset other than the validation set as the training set data. Some embodiments of this application determine whether to directly use the basic validation set as the validation set or to obtain the validation set based on the dataset to be processed by whether the amount of data contained in the basic validation set reaches the target data size required for the validation set, thereby improving the versatility of the technical solution.

[0066] It should be noted that the target data volume required for the validation set is determined based on the data volume of the dataset and the partition ratio between the dataset and the validation set. For example, if the dataset contains 1000 data points and the validation set is partitioned into quarters, then the target data volume required for the validation set is 1000 * 1 / 4 = 250. In some embodiments of this application, if the total data volume of the basic validation set is less than the target data volume required for the validation set, then the initial set of the dataset to be processed is the dataset itself, and the initial set of the validation set to be processed is empty; if the total data volume of the basic validation set is less than the target data volume required for the validation set, then the initial set of the dataset to be processed is the data included in the dataset excluding the basic validation set, and the initial set of the validation set to be processed is the basic validation set; if the total data volume of the basic validation set is greater than the target data volume required for the validation set, then the initial set of the dataset to be processed is the basic validation set, and the initial set of the validation set to be processed is empty.

[0067] S104, Obtain the training set based on the data in the dataset other than the validation set.

[0068] For example, in some embodiments of this application, the data in the dataset other than the validation set is used as the training set data. In some embodiments of this application, the data in the dataset other than the validation set also needs to undergo data augmentation processing before being used as the training set data.

[0069] The following example illustrates the process of obtaining the validation set based on the dataset to be processed and the validation set to be processed in step S103.

[0070] For example, in some embodiments of this application, the process of obtaining the verification set based on the dataset to be processed and the verification set to be processed in step S103 includes, for instance:

[0071] S171, Select K data points from the dataset to be processed.

[0072] like Figure 6As shown, some embodiments of this application employ principles for selecting K data points from the dataset to be processed, including a first principle 401, a second principle 402, and a third principle 403. The first principle 401 ensures that the selected K data points have as many label categories as possible. The second principle 402 ensures that the actual data volume corresponding to each label in the selected K data points is as close as possible to the target data volume of the corresponding label category. The third principle 403 ensures that the ratio of the total number of labels corresponding to the selected K data points to K, and the ratio of the total number of labels corresponding to the dataset to the total data volume corresponding to the dataset, are as close as possible. It should be noted that in some embodiments of this application, the first principle is quantified by label coverage, the second principle is quantified by class data imbalance (i.e., ensuring that the value of class data imbalance is as small as possible), and the third principle is quantified by label data imbalance (i.e., ensuring that the label data imbalance is as small as possible).

[0073] The following examples illustrate how some embodiments of this application select K data points.

[0074] In some embodiments of this application, the process of selecting K data points from the dataset to be processed in step S171 includes, for example, selecting the K data points from the dataset to be processed based on label coverage, class data imbalance, and label quantity imbalance. The label coverage is used to quantify the total number of label categories in the candidate data; the class data imbalance is used to quantify the difference between the actual data volume corresponding to each label category in the candidate data and the target data volume of the corresponding label category; and the label quantity imbalance is used to quantify the difference between a first ratio and a second ratio. The first ratio is the ratio of the total number of labels corresponding to the candidate data to the total data volume included in the candidate data; and the second ratio is the ratio of the total number of labels corresponding to the dataset to the total number of data points in the dataset. In other words, some embodiments of this application provide functions related to quantifying selection principles. These functions can better quantify and evaluate the K data points from each group of candidates, thereby selecting the K data points used to form the validation set.

[0075] The design principles of the label quantity imbalance function are illustrated below.

[0076] For example, in some embodiments of this application, the class data imbalance is the sum of the standardized differences between the actual data volume and the target data volume of each label category corresponding to any K selected data points, where the standardization is an axisymmetric function with any positive result. Some embodiments of this application provide design principles for the class data imbalance function, improving the versatility of the technical solution and enhancing the accuracy and objectivity of the obtained class data imbalance values.

[0077] For example, in some embodiments of this application, the label imbalance is the sum of the standardized differences between the total number of labels corresponding to any K selected data points and the second target label number. The standardization is an axisymmetric function with a positive result. The second target label number is the product of the ratio of the data size of the validation set to the total number of labels in the dataset and the total number of labels in the dataset. Some embodiments of this application provide design principles for the label imbalance function, improving the versatility of the technical solution and enhancing the accuracy and objectivity of the obtained label imbalance value.

[0078] The following examples illustrate how some embodiments of this application select the K data points from the dataset to be processed based on the aforementioned label coverage, category data imbalance, and label quantity imbalance.

[0079] For example, in some embodiments of this application, the process of selecting the K data points from the dataset to be processed based on label coverage, category data imbalance, and label quantity imbalance is as follows: Figure 7 As shown, the process includes, for example, the following steps: S1711, using the dataset to be processed as an initial candidate set; S1712, obtaining the top n combinations with the highest label coverage from the initial candidate set using an optimal solution search algorithm, to obtain a first candidate set, wherein one combination includes K candidate data; S1713, calculating the class imbalance of all combinations in the first candidate set and selecting the top m combinations in order, to obtain a second candidate set, wherein one combination includes K candidate data; S1714, calculating the label number imbalance of all combinations in the second candidate set and selecting the top k combinations in order, to obtain a third candidate set, wherein one combination includes K candidate data; S1715, selecting the first one from the third candidate set as the K data. Some embodiments of this application select K data from the dataset to be processed based on label coverage, class imbalance, and label number imbalance, such that the class imbalance of the selected K data is as small as possible, and the label number imbalance of the selected K data is also as small as possible, thereby improving the quality of the obtained validation set.

[0080] For example, in some embodiments of this application, selecting K data points from the dataset to be processed includes: selecting the K data points from the dataset to be processed according to a target normalization function, wherein the target normalization function is a function related to the label coverage, the class data imbalance, and the label quantity imbalance. Some embodiments of this application select K data points by defining a unified function. For example, the target normalization function is obtained by weighting a normalized label coverage function, a normalized class data imbalance function, and a normalized label quantity imbalance function. The normalized label coverage function is obtained by normalizing the label coverage, the normalized class data imbalance function is obtained by normalizing the class data imbalance, and the normalized label quantity imbalance function is obtained by normalizing the label quantity imbalance. Some embodiments of this application provide a design principle for the target normalization function, improving the quality of the validation set composed of K data points obtained according to this function.

[0081] S172, remove the K data points from the dataset to be processed and add the K data points to the verification set to be processed.

[0082] S173, determine whether the amount of data in the validation set to be processed is equal to the target amount of data required for the validation set. If yes, execute S174; otherwise, return to S171 and repeat the above process.

[0083] S174, take the obtained verification set to be processed as the first verification set, and align the number of tags in the first verification set to obtain the verification set.

[0084] In other words, the process of obtaining the verification set based on the dataset to be processed and the verification set to be processed in some embodiments of this application includes: selecting K data points from the dataset to be processed and removing the K data points from the dataset to be processed; adding the K data points to the verification set to be processed; if it is determined that the amount of data in the verification set to be processed is less than the target amount of data required by the verification set, repeating the above process until the amount of data in the verification set to be processed is equal to the target amount of data required by the verification set, and then using the verification set to be processed as the verification set.

[0085] Some embodiments of this application provide a process for selecting data from a dataset to be processed to form a validation set. In each selection, a portion of the data is selected according to the above selection principles to improve the final validation set by having as many label categories as possible, as little class label imbalance as possible, and as little label quantity imbalance as possible, thereby improving the quality of the trained model.

[0086] Some embodiments of the method for processing training data of the present application are exemplarily described below in conjunction with relevant calculation formulas.

[0087] Step 1: Divide the dataset into a training set and a validation set. Determine the ratio rate of the training set and the validation set to obtain the data volume of the validation set; then proceed to Step 2.

[0088] Step 2: First divide the validation set (generally, the data volume of the validation set is much smaller than that of the training set). Construct a basic validation set for all data: first perform random shuffle on all data in the dataset, then find one multi-label data for each type of label and add it to the basic validation set, and deduplicate the basic validation set; according to the data volume of the deduplicated basic validation set (denoted as BN), check whether it meets the data volume of the validation set (denoted as TN). If the volume requirement is met, jump to Step 7. If not, execute the following steps until reaching Step 7.

[0089] If the volume exceeds the data volume of the validation set, it needs to be reduced, that is, only TN documents are obtained from the basic validation set, the K value is recorded as equal to TN; the to-be-processed validation set is recorded as empty, and the to-be-processed dataset is recorded as the basic validation set.

[0090] If the volume does not exceed the data volume of the validation set, the basic validation set needs to be processed as follows first: only keep one piece of data under each label category (the purpose is to ensure that a) can be satisfied, and this solution can be selected in scenarios where a is very important) or clear the basic validation set (let Step 5 automatically select according to the algorithm) before performing subsequent data supplementation. Record the to-be-processed validation set as the basic validation set, record the to-be-processed dataset as the remaining data of the dataset excluding the to-be-processed validation set, and record the data volume of the to-be-processed validation set as VN.

[0091] Step 4: Select K pieces of data from the to-be-processed dataset (if K is not specified in the previous process, its value range is 1<K<TN-VN) and record them as KG (each piece of data corresponds to multiple labels). KG needs to satisfy the following a), b) and c):

[0092] a) The label coverage rate (the total number of label categories) is as large as possible.

[0093] b) The category data imbalance degree (group the data under KG by labels, denoted as TG, then count the corresponding data volume of each label, and calculate the sum of the gaps in the corresponding data volume of each label) is as small as possible.

[0094] For example, grouping the data under KG according to labels as TG, and then counting the number of data points corresponding to each label, is an example: KG corresponds to three data points: K1, K2, and K3. Data K1 corresponds to labels a and b, denoted as K1(a,b); data K2 corresponds to labels b and c, denoted as K2(b,c); and data K3 corresponds to label c, denoted as K3(c). Then TG is: a(K1), b(K1,K2), and c(K2,K3), where a(K1) indicates that label A includes data K1, b(K1,K2) indicates that label b includes data K1 and data K2, and c(K2,K3) indicates that label c includes data K2 and data K3. The number of data points corresponding to each label is as follows: label a has 1 data point, label b has 2 data points, and label c has 2 data points.

[0095] c) The label imbalance (the difference between the sum of all labels in KG and the sum of all data in KG and the difference between the sum of all labels in the dataset and the sum of all data in the dataset) should be as small as possible.

[0096] Remove the aforementioned K data points from the dataset to be processed (forming a new dataset to be processed) and add them to the validation set to be processed (forming a new validation set to be processed).

[0097] If the number of validation sets to be processed does not reach the target validation set data volume, repeat this step; otherwise, proceed to step 6.

[0098] The following exemplifies how embodiments of this application ensure that the above (a), (b), and (c) are satisfied.

[0099] The explanation for data balance in b) is: the proportion of samples within each label class should remain consistent with the proportion of samples within the corresponding label class in the original dataset. For example: the original dataset has two label classes, A (16 samples, i.e., 16 data points) and B (8 samples, i.e., 8 data points); the validation set split ratio is 1 / 4 (i.e., taking 1 / 4 of the original dataset as the validation set). Then, the target label distribution is: label A (4 samples, i.e., 4 data points) and label B (2 samples, i.e., 2 data points), meaning the overall proportion of the validation set within each label class should be maintained (1 / 4). The explanation for label imbalance in c) is: the overall proportion of labeled data should remain consistent with the original dataset. For example, the original dataset contains four documents: document D1 (16 labels), document D2 (8 labels), document D3 (5 labels), and document D4 (4 labels). The ratio for splitting the validation set is 1 / 4 (i.e., taking 1 / 4 of the original dataset as the validation set). Therefore, the ideal split would result in the validation set having 1 / 4 = 32 / 4 = 8 labels (as an example of the second target label count). Thus, D2 is considered the optimal result.

[0100] It should be noted that the methods to achieve the above conditions a) and b) are as follows:

[0101] First, define a unified measurement metric:

[0102] a) Tag coverage is used to define the total number of tag categories. An example function for tag coverage is defined as follows:

[0103]

[0104] Among them, C i This refers to the number of distinct label categories corresponding to each data point in KG (i.e., the K selected data points).

[0105] b) Class imbalance

[0106] For a single KG, the sum of the standardized differences between the actual number of data points and the target number of data points under each TG can be achieved using any axisymmetric function with a positive result, such as x. 2 , |x|, e x The example function definition for class imbalance is as follows:

[0107]

[0108] Where M is the total number of distinct labels in KG (KG is a set of K data), and D i DT represents the actual number of data items under each label in KG within TG. i This represents the number of target data items under this category of tags.

[0109] c) Tag quantity imbalance

[0110] For a single KG, the sum of the standardized differences between the number of all tags and the number of target tags can be achieved using any axisymmetric function that produces a positive result, such as x. 2 , |x|, e x The example function definition for an imbalance in the number of tags is as follows:

[0111] tagnum_unbalance(KG) = (L - LT) 2

[0112] Where L represents the total number of labels corresponding to any K data points, and LT represents the value of the second target label count (the calculation method for this value can be found in the explanation above). In other words, the value of the second target label count is the product of the ratio of the data size of the validation set to the total data size of the dataset and the total number of labels in the dataset.

[0113] The following two examples illustrate how to select K data points based on the above calculation formula:

[0114] Example 1

[0115] The specific process of solving for the value of the above function in sequence includes the following steps:

[0116] The initial candidate set is the dataset to be processed.

[0117] a) For the above candidate set, by using the optimal solution search algorithm not limited to beamsearch, the top n combinations with the highest tag coverage (i.e., coverage (KG) value) (each combination contains at least K documents) can be obtained as the candidate set that meets the conditions.

[0118] b) For the above candidate set, calculate the class imbalance (i.e., unbalance (KG) value) of all combinations, and sort them in order (i.e., in ascending order) and take the top n combinations (containing K documents) as the candidate set that meets the conditions.

[0119] c) For the above candidate set, calculate the tag number imbalance (i.e., tagnum_unbalance(KG) value) of all combinations, and take the top n combinations (containing K documents) as the candidate set that meets the conditions.

[0120] d) Select the first candidate from the candidate set as the single-best result that satisfies the conditions.

[0121] It should be noted that the order of the three steps a to c above can be arbitrarily interchanged.

[0122] Example 2

[0123] A unified objective function is defined and solved using an optimal solution search algorithm. In some embodiments of this application, different normalized objective functions are designed for conditions a), b), and c), and the results are weighted to obtain the objective normalization function. For example... Figure 8 As shown, in some embodiments of this application, the method for obtaining K data points includes: S1716, constructing a normalization objective function; S1717, selecting K data points from the dataset to be processed according to the normalization objective function. The following first explains how to construct the normalization objective function:

[0124] a) An exemplary formula for the tag coverage normalization function (i.e., the normalized tag coverage function) is as follows:

[0125]

[0126] Where |D| is the total number of all label types.

[0127] b) An exemplary formula for the class imbalance normalization function (i.e., the normalized class imbalance function) is as follows:

[0128]

[0129] Where i is an integer; n is the total number of KGs to be solved (corresponding to the n candidate KGs in each step of the optimal solution algorithm, such as the beam width in beam search).

[0130] c) An example formula for the label imbalance normalization function (i.e., the normalized label imbalance function) is as follows:

[0131]

[0132] Where i is an integer; n is the total number of KGs to be solved (corresponding to the n candidate KGs in each step of the optimal solution algorithm, such as the beam width in beam search).

[0133] The final score function for each KG (as an example of the objective normalization function) is:

[0134] score(KG i = wa / coverage_norm(KG) i )+wb*unbalance_norm(KG i )+

[0135] wc*tagnum_unbalance_norm(KG i )

[0136] Among them, wa, wb, and wc are weight parameters, and their values ​​are determined according to the actual use case.

[0137] Using an optimal solution search algorithm not limited to beamsearch, at each step from 1 to K, according to the above score function, the score is calculated using the score(KG). i The score is calculated, and the KG with the smallest score in the final step (step K) is taken as the single optimal result that satisfies the conditions.

[0138] It should be noted that, in some embodiments of this application, the above-mentioned target normalization function score(KG) is used. i It can also be transformed into the following score'(KG) i The function then calculates the top k largest results, taking the KG with the highest score as the single-run optimal result that satisfies the conditions.

[0139] 1.score'(KG i ) = 1 / score(KGi ) or -score(KG i )wait

[0140] 2.score'(KG i = wa*coverage_norm(KG i )+wb / unbalance_norm(KG i )+

[0141] wc / tagnum_unbalance_norm(KG i )

[0142] Step 6: After aligning the number of labels in the validation set (i.e., ensuring that the number of labels in the validation set is consistent with the number of labels in the target validation set), the validation set is obtained. The remaining data in the dataset excluding the validation set is set as the training set, and the standardization is completed.

[0143] Step 7: Perform label statistics on all data outside the validation set to identify rare labels (labels that are too few in number or whose proportion of the total number of labels is too small). Then, resample the data with rare labels to construct the validation set. For example, sampling enhancement algorithms are commonly used, including but not limited to SMOTE (Synthetic Minority Oversampling Technique). Figure 4 As shown in step S190, the data in the training set is augmented, and the augmented data is used as the final training set, thus obtaining the training set and the validation set.

[0144] It should be noted that the ratio of training to validation sets (rate) and beam width (B) mentioned above are hyperparameters. Generally, the rate is set to 9:1 or 8:2 to ensure the best possible training and validation results. Beam widths (B1 and B2) are generally set to around 5 to allow beam search to obtain better results. The iterative sampling method described above solves the standardization problem of multi-label datasets. By progressively determining the validation set partition, the label categories and proportions of the training and validation sets are ensured to be as close as possible to the overall data, guaranteeing the rationality of dataset standardization. At the same time, it can make full use of the data, relatively improving the capabilities of the model system, and the trained model is more robust.

[0145] The technical solution of this application is illustrated below with a specific application scenario as an example.

[0146] Published articles often require natural language processing (NLP) models for text classification to facilitate search and recommendation. Many articles cover multiple fields, necessitating the categorization of these articles into various classes. For NLP models handling multi-labeled text, training data is typically multi-labeled datasets. Due to requirements such as text format and word count, the proportions of different labels in multi-labeled text classification datasets often vary significantly. For instance, in portal website news data, the proportion of articles categorized as society, technology, and finance is much higher than that of articles categorized as religion and law. Furthermore, certain specific labels may exhibit strong correlations; for example, multi-labeled articles in portal website news that generally contain educational content are also likely to contain social content.

[0147] If the training and validation sets are randomly selected, there will be a significant difference between the number of label categories and their proportions. This can cause the model to become overly sensitive to certain categories during training, thus reducing its performance.

[0148] Some embodiments of this application can be adopted. Figure 3 The steps involved using Beam Search to extract more validation set data in appropriate proportions, ensuring that the labels of the training and validation sets were not lacking in categories and that the proportions were close to the overall data, thus avoiding model prediction errors due to data issues.

[0149] Some embodiments of this application provide a method for training a natural language processing system, the method comprising: training the natural language processing system using a training set and a validation set obtained according to any of the embodiments of the above-described method for processing training data.

[0150] Some embodiments of this application provide a computer storage medium having a computer program stored thereon, which, when executed by a processor, can implement the methods included in the above-described method for processing training data.

[0151] Please refer to Figure 9 , Figure 9 The apparatus for processing training data provided in the embodiments of this application is shown. It should be understood that this apparatus is similar to the one described above. Figure 3 Corresponding to the method embodiments, it can execute the various steps involved in the above method embodiments. The specific functions of the device can be found in the description above. To avoid repetition, detailed descriptions are appropriately omitted here. The device includes at least one software function module that can be stored in the memory or embedded in the device's operating system in the form of software or firmware. The device for processing training data includes: a category label set acquisition module 901, a basic validation set acquisition module 902, a validation set acquisition module 903, and a training set acquisition module 904.

[0152] The category label set acquisition module 901 is configured to retrieve the labels of each category from the dataset to obtain the category label set.

[0153] The basic validation set acquisition module 902 is configured such that each label in the category label set selects a multi-label data from the dataset, and uses the category label set and all selected multi-label data as the basic validation set.

[0154] The validation set acquisition module 903 is configured to obtain the validation set based on the size relationship between the total amount of data included in the basic validation set and the target amount of data required for the validation set.

[0155] The training set acquisition module 904 is configured to obtain a training set based on data in the dataset other than the validation set.

[0156] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the aforementioned method, and will not be elaborated further here.

[0157] like Figure 10 As shown, some embodiments of this application provide an electronic device 500, including a memory 510, a processor 520, and a computer program stored on the memory 510 and executable on the processor 520, wherein when the processor 520 reads the program from the memory 510 via a bus 530 and executes the program, it can implement the methods described in any embodiment of the method for processing training data.

[0158] Processor 520 can process digital signals and can include various computing architectures. For example, it can be a complex instruction set computer architecture, a reduced instruction set computer architecture, or an architecture that implements multiple instruction set combinations. In some examples, processor 520 can be a microprocessor.

[0159] The memory 510 can be used to store instructions executed by the processor 520 or data related to the execution of instructions. These instructions and / or data may include code used to implement some or all of the functions of one or more modules described in the embodiments of this application. The processor 520 of the embodiments of this disclosure can be used to execute the instructions in the memory 510 to implement… Figure 3 The method shown. Memory 510 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0160] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0161] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0162] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0163] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0165] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method for processing training data, characterized in that, The method includes: The labels for each category are obtained from the dataset to obtain a set of category labels; the dataset is a multi-label text dataset of news from a portal website, and the labels are category labels. For each label in the category label set, select one multi-label data point from the dataset, and use the category label set and all selected multi-label data points as the basic validation set; the multi-label data point is news text data with multi-column category labels; If it is confirmed that the total amount of data in the basic validation set is not equal to the target amount of data required for the validation set, then the validation set is obtained based on the dataset to be processed and the validation set to be processed. The initial set of the dataset to be processed is the dataset or a dataset determined by the basic validation set. The initial set of the validation set to be processed is an empty set or the basic validation set. The training set is obtained from the dataset excluding the validation set; the training set and validation set are used to train the natural language processing system.

2. The method as described in claim 1, characterized in that, If the total amount of data in the basic validation set is less than the target amount of data required by the validation set, then the initial set of the dataset to be processed is the data included in the dataset excluding the basic validation set, and the initial set of the validation set to be processed is the basic validation set. If the total amount of data included in the basic validation set is greater than the target amount of data required by the validation set, then the initial set of the dataset to be processed is the basic validation set and the initial set of the dataset to be processed is empty.

3. The method as described in any one of claims 2, characterized in that, The step of obtaining the validation set based on the dataset to be processed and the validation set to be processed includes: Select K data points from the dataset to be processed; Add the K data points to the verification set to be processed; If it is determined that the amount of data in the verification set to be processed is less than the target amount of data required for the verification set, then the above process is repeated until the amount of data in the verification set to be processed is equal to the target amount of data required for the verification set, at which point the verification set to be processed is taken as the verification set.

4. The method as described in claim 3, characterized in that, The step of selecting K data points from the dataset to be processed includes: The K data points are selected from the dataset to be processed based on label coverage, category data imbalance, and label quantity imbalance. The label coverage is used to quantify the total number of label categories in the candidate data. The category data imbalance is used to quantify the difference between the actual data volume corresponding to each label in the candidate data and the target data volume of the corresponding label. The label quantity imbalance is used to quantify the difference between a first ratio and a second ratio. The first ratio is the ratio of the total number of labels corresponding to the candidate data to the total amount of data included in the candidate data. The second ratio is the ratio of the total number of labels corresponding to the dataset to the total number of data in the dataset.

5. The method as described in claim 4, characterized in that, The class data imbalance is the sum of the differences between the actual data volume of each class label and the target data volume of the corresponding class label for any K selected data points after standardization. The standardization is an axisymmetric function with any positive result.

6. The method as described in claim 5, characterized in that, The formula for calculating the imbalance of the category data is: unbalance(KG) Where M represents the total number of label categories corresponding to any K data points. This represents the actual amount of data contained in the i-th type of label. The target data volume corresponding to the i-th label is determined based on the proportion of the validation set in the dataset.

7. The method as described in claim 4, characterized in that, The label imbalance is the sum of the differences between the number of all labels corresponding to any K selected data points and the number of the second target labels, after standardization. The standardization belongs to any axisymmetric function with a positive result. The second target label number is the product of the ratio of the data volume of the validation set to the data volume of the dataset and the total number of labels in the dataset.

8. The method as described in claim 7, characterized in that, The formula for the label quantity imbalance is as follows: tagnum_unbalance(KG)= Where L represents the total number of all labels corresponding to any K data points, and LT represents the value of the second target label count.

9. The method as described in claim 4, characterized in that, The step of selecting the K data points from the dataset to be processed based on label coverage, category data imbalance, and label quantity imbalance includes: The dataset to be processed is used as the initial candidate set; The first candidate set is obtained by using an optimal solution search algorithm to find the top n combinations with the highest label coverage from the initial candidate set, where each combination includes K candidate data. Calculate the class imbalance of all combinations in the first candidate set, sort them in order, and take the top n combinations to obtain the second candidate set, where each combination includes K candidate data. Calculate the label imbalance of all combinations in the second candidate set, sort them in order, and take the top k combinations to obtain the third candidate set, where each combination includes K candidate data. The first one from the third candidate set is selected as the K data.

10. The method as described in claim 4, characterized in that, The step of selecting K data points from the dataset to be processed includes: selecting the K data points from the dataset to be processed according to a target normalization function, wherein the target normalization function is a function related to the label coverage, the class data imbalance, and the label quantity imbalance.

11. The method as described in claim 10, characterized in that, The target normalization function is obtained by weighting the normalized label coverage function, the normalized category data imbalance function, and the normalized label quantity imbalance function. The normalized label coverage function is obtained by normalizing the label coverage, the normalized category data imbalance function is obtained by normalizing the category data imbalance, and the normalized label quantity imbalance function is obtained by normalizing the label quantity imbalance.

12. A method for training a natural language processing system, characterized in that, The method includes: training the natural language processing system using the training set and validation set obtained according to any one of the methods in claims 1-11.

13. A computer storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it can implement the method described in any one of claims 1-12.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the program, it can implement the method as described in any one of claims 1-12.

Images