Training set processing method and device for classification, electronic equipment and storage medium

Abstract

The application provides a training set processing method and device for classification, electronic equipment and a storage medium, comprising: obtaining a classification sample set; determining each prediction classification corresponding to each sample, the probability of each prediction classification, and the disambiguation score corresponding to each sample; for each sample, determining the most similar sample corresponding to the sample from each sample having a target prediction classification corresponding to the sample; based on the disambiguation score, determining the most similar prediction classification corresponding to each classification label, and based on the most similar prediction classification, determining a first target sample in which the prediction classification is the most similar prediction classification from all samples having the classification label, and a second target sample in which the prediction classification is the classification label from all samples having the most similar prediction classification; by analyzing the first target sample, the second target sample and the most similar sample corresponding to each sample, defects such as confusion of categories or labeling errors of samples can be quickly identified, and the sample cleaning speed is improved.

Description

Technical Field

[0001] This invention relates to the field of text classification technology, and more specifically, to a training set processing method, apparatus, electronic device, and storage medium for classification. Background Technology

[0002] Classification tasks play a significant role in many scenarios, such as intent recognition in intelligent customer service and identification of plants and animals in nature. In addition to relying on classification model technology, classification tasks also heavily depend on the classification corpus. The classification effect is closely related to the data quality and sample quantity of the training set. When there are a large number of labeling errors in the labeled samples of the training set, the performance of the algorithm will be affected, which will seriously affect the classification effect. Summary of the Invention

[0003] One objective of this invention is to provide a training set processing method, apparatus, electronic device, and storage medium for classification, which can quickly clean the training set and identify samples or categories with incorrect or confused labeling. Embodiments of this invention can be implemented as follows:

[0004] In a first aspect, the present invention provides a training set processing method for classification, the method comprising:

[0005] Obtain a classification sample set; the classification sample set contains multiple samples and a classification label corresponding to each sample; one classification label corresponds to at least one sample;

[0006] Determine each predicted category corresponding to each sample, the probability of each predicted category, and the disambiguation score corresponding to each sample; wherein the disambiguation score characterizes the degree of error of the classification label corresponding to the sample;

[0007] For each sample, from all samples with the target predicted classification corresponding to the sample, determine the most similar sample corresponding to each sample, where the probability of the target predicted classification is the highest;

[0008] Based on the disambiguation score, the most similar predicted category corresponding to each category label is determined. Based on the most similar predicted category, a first target sample predicted to be classified as the most similar predicted category is determined from all samples with the category label, and a second target sample predicted to be classified as the category label is determined from all samples with the most similar predicted category.

[0009] Wherein, the first target sample and the second target sample are used to indicate the implementation of a merging strategy for the classification label and the most similar predicted classification, or to indicate the implementation of a correction strategy for the first target sample and / or the second target sample; the most similar sample is used to indicate the correction of the sample.

[0010] Secondly, the present invention provides a training set processing apparatus for classification, comprising:

[0011] The acquisition module is used to acquire a classification sample set; the classification sample set contains multiple samples and a classification label corresponding to each sample; one classification label corresponds to at least one sample;

[0012] The determination module is used to determine each predicted category corresponding to each sample, the probability of each predicted category, and the disambiguation score corresponding to each sample; wherein the disambiguation score represents the error score of the classification label corresponding to the sample;

[0013] The determining module is further configured to, for each sample, determine the most similar sample corresponding to each sample from among the samples having the target prediction classification corresponding to the sample, wherein the probability of the target prediction classification is the highest; the probability of the target prediction classification is the highest.

[0014] The determining module is further configured to determine the most similar predicted category corresponding to each category label based on the disambiguation score, and based on the most similar predicted category, determine a first target sample predicted to be classified as the most similar predicted category from all samples with the category label, and determine a second target sample predicted to be classified as the category label from all samples with the most similar predicted category.

[0015] Wherein, the first target sample and the second target sample are used to indicate the implementation of a merging strategy for the classification label and the most similar predicted classification, or to indicate the implementation of a correction strategy for the first target sample and / or the second target sample; the most similar sample is used to indicate the correction of the sample.

[0016] Thirdly, the present invention provides an electronic device including a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the training set processing method for classification provided in the first aspect.

[0017] Fourthly, the present invention provides a readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training set processing method for classification as provided in the first aspect.

[0018] This invention provides a training set processing method, apparatus, electronic device, and storage medium for classification. The method includes: firstly, obtaining a classification sample set; then, predicting the category of each sample and the probability of each category. Since each sample has a correct classification label, for each category, the predicted category with the greatest conflict with that category, the first target sample corresponding to that predicted category, and the second target sample whose predicted result is that category can be determined from the predicted results of samples with that predicted category. By analyzing the first and second target samples, it can be determined whether categories need to be merged or whether samples need to be adjusted. It can also determine the most similar sample corresponding to each sample, and then determine whether the category of the sample needs to be adjusted. The whole process can quickly identify labeling errors and labeling confusion, improving the sample cleaning speed. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present invention;

[0021] Figure 2 A structural block diagram of an electronic device provided in an embodiment of the present invention;

[0022] Figure 3 A schematic flowchart illustrating a training set processing method for classification provided in an embodiment of the present invention;

[0023] Figure 4 This is a schematic diagram of a text sample set in an embodiment of the present invention;

[0024] Figure 5 A schematic diagram illustrating the predicted classification provided in an embodiment of the present invention;

[0025] Figure 6 This is a schematic flowchart of step S302 in an embodiment of the present invention;

[0026] Figure 7 This is a schematic diagram of sample knowledge corresponding to each sample provided in the embodiments of the present invention;

[0027] Figure 8 A schematic flowchart of step S304 provided in an embodiment of the present invention;

[0028] Figure 9 This is a schematic diagram of category disambiguation result analysis provided in an embodiment of the present invention;

[0029] Figure 10 This is a functional block diagram of a training set processing device for classification provided in an embodiment of the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0031] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0032] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0033] In the description of this invention, it should be noted that if terms such as "upper," "lower," "inner," or "outer" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0034] Furthermore, the terms "first" and "second" are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0035] It should be noted that, where there is no conflict, the features in the embodiments of the present invention can be combined with each other.

[0036] Classification tasks are useful in many scenarios, such as intent recognition in intelligent customer service and identification of plants and animals in nature. For an example of text classification, please refer to [link to relevant documentation]. Figure 1 , Figure 1This is a schematic diagram of an application scenario provided by an embodiment of the present invention. The scenario includes a terminal device 11 and a server 12. Various applications can be installed on the terminal device 11. After the application on the terminal device 11 and the server 12 establish a communication connection through a communication network, the client on the terminal device 11 can send the text to be recognized to the server 12. The server 12 classifies the text, obtains the classification result, and then sends the classification result back to the application on the terminal device 11.

[0037] Among them, the terminal device 11 may be, but is not limited to, a computer device with information collection function such as a personal computer, laptop, smartphone, tablet computer, smart wearable device, etc.

[0038] Server 12 can be implemented using a standalone server, a server cluster composed of multiple servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Terminal device 11 and server 12 can be connected directly or indirectly via wired or wireless communication; this application does not impose any restrictions on this connection.

[0039] Terminal device 11 and server 12 can communicate with each other through a communication network. The communication network can be a wireless fidelity (Wi-Fi) hotspot network, Bluetooth (BT) network, or near field communication (NFC) network, or it can be a third-generation wireless telephone technology (3G) network, a fourth-generation mobile communication technology (4G) network, a fifth-generation mobile communication technology (5G) network, a future public land mobile network (PLMN), or the Internet.

[0040] See also Figure 1Currently, before performing a classification task, a classification model needs to be trained using a pre-labeled training set, and then the trained model is used to perform classification. It can be seen that the classification performance is highly dependent on the quality of the training set data. When the labeled samples in the training set contain errors—for example, some samples are incorrectly labeled, or there are conflicts between certain categories—the performance of the classification model will be affected, severely impacting the classification results.

[0041] Therefore, this invention provides a training set processing method that first cleans the training set to achieve knowledge disambiguation. Here, "disambiguation" refers to eliminating sample labeling errors and class confusion. The results are then fed back to the operators, who can adjust the samples or classes to achieve the effect of pre-cleaning the training set.

[0042] Please see first. Figure 2 , Figure 2 This is a structural block diagram of an electronic device provided in an embodiment of the present invention. The electronic device can be used to execute the training set processing method for classification provided in an embodiment of the present invention.

[0043] like Figure 2 As shown, the electronic device 200 includes a memory 201, a processor 202, and a communication interface 203. The memory 201, processor 202, and communication interface 203 are electrically connected to each other directly or indirectly to realize data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines.

[0044] The memory 201 can be used to store software programs and modules, such as the instructions / modules of the training set processing device 400 for classification provided in this embodiment of the invention. These can be stored in the memory 201 in the form of software or firmware, or embedded in the operating system (OS) of the electronic device 200. The processor 202 executes various functional applications and data processing by executing the software programs and modules stored in the memory 201. The communication interface 203 can be used for signaling or data communication with other node devices.

[0045] The memory 201 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0046] Processor 202 can be an integrated circuit chip with signal processing capabilities. Processor 202 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0047] Understandable. Figure 2 The structure shown is for illustrative purposes only; the electronic device 200 may also include more than [other components]. Figure 2 The more or fewer components shown, or having the same Figure 2 The different configurations shown. Figure 2 The components shown can be implemented using hardware, software, or a combination thereof.

[0048] The following is based on Figure 2 The electronic device shown is the execution entity. A detailed description of the training set processing method for classification provided in this embodiment of the invention is available in the following text. Figure 3 , Figure 3 This is a schematic flowchart of a training set processing method for classification provided in an embodiment of the present invention. The method includes:

[0049] S301: Obtain the classification sample set; the classification sample set contains multiple samples and the classification label corresponding to each sample; one classification label corresponds to at least one sample;

[0050] S302: Determine the predicted categories for each sample, the probability of each predicted category, and the disambiguation score for each sample; where the disambiguation score characterizes the degree of error of the classification label corresponding to the sample.

[0051] S303: For each sample, from all samples with the target prediction classification corresponding to the sample, determine the most similar sample corresponding to each sample, which has the highest probability of target prediction classification;

[0052] S304: Based on the disambiguation score, determine the most similar predicted class corresponding to each class label, and based on the most similar predicted class, determine the first target sample predicted to be the most similar predicted class from all samples with class labels, and determine the second target sample predicted to be the class label from all samples with the most similar predicted class.

[0053] The first target sample and the second target sample are used to indicate the implementation of a merging strategy for the classification label and the most similar predicted classification, or to indicate the implementation of a correction strategy for the first target sample and / or the second target sample; the most similar sample is used to indicate the correction of the sample.

[0054] In the training set processing method described above, a classification sample set is first obtained, and then the category of each sample and the probability of each category are predicted. Since each sample has a correct classification label, for each category, the predicted category with the greatest conflict with that category, the first target sample corresponding to that predicted category, and the second target sample whose predicted result is that category can be determined from the prediction results of the samples with that predicted category. By analyzing the first and second target samples, it can be determined whether the categories need to be merged or whether the samples need to be adjusted. It can also determine the most similar sample corresponding to each sample, and then determine whether the category of the sample needs to be adjusted. The whole process can quickly identify labeling errors and labeling confusion, thus improving the sample cleaning speed.

[0055] The following is a detailed description of steps S301 to S304.

[0056] In step S301, the classification sample set is obtained.

[0057] In this embodiment, the classification sample set contains multiple samples and a classification label corresponding to each sample; one classification label corresponds to at least one sample.

[0058] As an optional implementation, the samples in the classification sample set can be corpora, text, images, etc., and there is no limitation here.

[0059] For example, if the sample is text, please refer to [link / reference]. Figure 4 , Figure 4 This is a schematic diagram of a text sample set in an embodiment of the present invention, where sentence is a sample, classify is the category corresponding to the sample, label, and split_num is a sample identifier. The sample segmentation identifier in this embodiment of the present invention is used as the data basis for subsequent cross-training of the classified sample set.

[0060] As an optional implementation method, the application software can be crawled in the background by data crawling, or the user's behavior records in the application software can be recorded in real time, and the text input by the user can be extracted to form a classification sample set. The application software can be, but is not limited to, media applications, game applications, shopping applications, etc., and is not limited here.

[0061] It is understandable that the classification labels are manually labeled. The classification label corresponding to each sample may match the actual category of the sample, that is, the classification label is a correct classification, or the classification label does not match the actual category of the sample, that is, the classification label is not a correct classification. The embodiments of the present invention can help operators identify incorrectly labeled samples and easily confused categories.

[0062] In step S302, the predicted categories corresponding to each sample, the probability of each predicted category, and the disambiguation score corresponding to each sample are determined.

[0063] In this embodiment, the classification model in this invention can be used to predict each sample to obtain the predicted classification and the probability of the predicted classification for each sample. It can be understood that the probability refers to the probability of the predicted classification being the classification label corresponding to the sample.

[0064] The disambiguation score mentioned above can be used to roughly estimate the degree of error of the classification label corresponding to the sample. In this embodiment, the disambiguation score is the maximum value between the maximum misclassification probability and the confusion probability. The maximum misclassification probability is the probability corresponding to the predicted classification that is different from the classification label but has the highest probability. The confusion probability is the difference between the two highest probabilities among all probabilities. In this way, the embodiment of the present invention obtains the disambiguation score of the sample through the above two indicators, which can target both labeling errors and confused labels, providing a data basis for subsequent identification of whether there are labeling errors and labeling confusion in the sample.

[0065] See Figure 5 , Figure 5 This is a schematic diagram of the predicted classification provided in an embodiment of the present invention. Figure 5 The system displays the top two predicted categories and probabilities for each sentence. In practical implementations, the number of predicted categories to be displayed can be customized, and it is not limited to displaying the top two predicted categories with probabilities. The prediction results for all samples can be obtained through the above method.

[0066] Therefore, this embodiment also provides an implementation method for step S302, please refer to [link / reference]. Figure 6 , Figure 6 Here is a schematic flowchart of step S302 in an embodiment of the present invention:

[0067] S302-1: Using the trained classification model, obtain the predicted classification for each sample and the probability of each predicted classification.

[0068] In this embodiment, the classification model can be trained using any existing training method.

[0069] As one optional implementation method, the training method of the classification model in this embodiment may include the following steps:

[0070] a1: Based on the sample identifier corresponding to each sample, determine multiple sample subsets from the training set and the prediction sample set corresponding to each sample subset.

[0071] The union of each sample subset and the corresponding predicted sample set is the classification sample set. Within a sample subset, the remainder obtained by dividing the sample identifier of each sample by the number of sample subsets is the same, and the remainder is different from the remainder corresponding to each sample in the predicted sample set.

[0072] In this embodiment, the sample identifier of each sample is divided by the number of sample subsets to obtain the remainder corresponding to each sample. Then, samples with the same remainder are grouped into a sample subset, and samples with other remainders are grouped into the prediction sample set corresponding to that sample subset.

[0073] For ease of understanding, let's assume the sample subset has 3 subsets, with remainders set to 0, 1, and 2. The classification sample set is divided into 3 subsets. The first subset, denoted as S, consists of samples where the remainder after dividing "split_num" by 3 is 1 or 2. 1，2 Then the samples with a remainder of 0 are considered as S. 1，2 The first set of predicted samples; the second subset of samples with remainders of 0 or 1, denoted as S. 0，1 Then the samples with a remainder of 2 are considered as S. 0，1 The predicted sample set of the sample subset; samples with remainders of 0 and 2 form a third sample subset, denoted as S. 0，2 Then the samples with a remainder of 1 are considered as S. 0，2 The predicted sample set.

[0074] In other words, in the actual implementation process, we can first determine the number N of sample subsets, then determine N different remainders that are the same as the number of sample subsets, and then in the process of determining the sample subsets, we first form a sample subset with N-1 remainders, and form a predicted sample set for the sample subset with the other remainder, and so on, to obtain multiple sample subsets and the predicted sample set corresponding to each sample subset.

[0075] a2: Train the classification model using each sample subset in sequence, and use the trained classification model to predict the prediction sample set corresponding to the sample subset, to obtain the prediction class of each prediction sample in the prediction sample set, and the probability of each prediction class.

[0076] S302-2: For each sample, based on the comparison between the predicted classification and the corresponding classification label of the sample, determine the maximum misclassification probability and confusion probability of the sample.

[0077] The purpose of comparing the predicted classification with the corresponding classification label of the sample is to determine whether there is a predicted classification that is the same as the classification label. Different comparison results can provide different guidance for determining the maximum misclassification probability.

[0078] In this embodiment, the maximum misclassification probability refers to the probability of misclassifying a sample (i.e., different from the classification label). For example, assuming a sample of class A, the predicted classification results are class A, class B, and class C. Then, class B and class C can both be regarded as misclassifications of class A samples. The probability of misclassification with the highest probability is taken as the maximum misclassification probability, that is, the probability of class B.

[0079] The confusion probability is the probability that there is a conflict between the predicted categories of a sample. For example, continuing with the above example, the relationship between the probabilities of the predicted categories A, B, and C is P. A >P B >P C The confusion probability is the difference between the probabilities of class A and class B. The confusion probability can be used to measure how easily class A and class B are confused.

[0080] Therefore, this embodiment of the invention provides an implementation method for step S302-2, namely:

[0081] b1: Determine the first and second maximum probabilities in descending order of probability, and use the difference between the first and second maximum probabilities as the confusion probability.

[0082] b2: If the predicted classification corresponding to the first maximum probability is consistent with the classification label, then the second maximum probability is taken as the maximum misclassification probability.

[0083] b3: If the predicted category corresponding to the first maximum probability is inconsistent with the category label, then the first maximum probability is taken as the maximum misclassification probability.

[0084] S302-3: The maximum value between the maximum misclassification probability and the confusion probability is determined as the disambiguation score corresponding to the sample.

[0085] By using the maximum misclassification probability and confusion probability obtained above, the disambiguation score corresponding to each sample can be determined. Then, the category disambiguation analysis results and sample disambiguation analysis results can be compiled and output. The sample disambiguation analysis results will be introduced first below.

[0086] Please see Figure 7 , Figure 7 This is a schematic diagram of sample knowledge corresponding to each sample provided in the embodiments of the present invention, from... Figure 7 As can be seen from the data, for each sample, the embodiments of the present invention can obtain the disambiguation score corresponding to the sample, as well as the most similar category corresponding to the category of the sample, the probability of the most similar category, the confusion probability, etc. Based on this sample knowledge, the embodiments of the present invention can continue to explore and obtain categories that may have labeling errors or are prone to confusion.

[0087] In step S303, for each sample, the most similar sample corresponding to the target prediction classification is determined from all samples with the target prediction classification corresponding to the sample, and the sample with the highest probability of target prediction classification is determined.

[0088] In this embodiment, each sample has multiple predicted categories. The predicted category with the highest probability is taken as the target predicted category. It should be noted that the predicted category with the highest probability may be the same as or different from the category label of the sample. Therefore, the most similar sample can be determined based on the two different situations.

[0089] In the first case: if the target predicted classification is consistent with the classification label corresponding to the sample, then the output of the most similar sample is empty;

[0090] The second scenario: If the target predicted classification is inconsistent with the classification label corresponding to the sample, extract all candidate samples with the target predicted classification from the classification sample set, calculate the similarity between all candidate samples and the sample, and determine the candidate sample with the highest similarity as the most similar sample.

[0091] In this embodiment, all samples in the classification sample set can be sorted in descending order of disambiguation scores, and then the most similar samples of each sample can be displayed in turn, which facilitates the analysis of possible reasons for prediction errors.

[0092] The results of the category disambiguation analysis will be presented below.

[0093] In step S304, based on the disambiguation score, the most similar predicted category corresponding to each category label is determined, and based on the most similar predicted category, a first target sample with the most similar predicted category is determined from all samples with category labels, and a second target sample with the most similar predicted category is determined from all samples with the most similar predicted category.

[0094] In this embodiment, one classification label corresponds to at least one sample. That is, assuming there are M samples in the classification sample set and the corresponding number of categories is N, then N and M satisfy the relationship that N is less than or equal to M. For each category, this embodiment of the invention can find the most similar category with the greatest conflict. The greatest conflict can be understood as the two categories cannot be easily distinguished. The reason for this situation may be that the granularity of category division is too small.

[0095] Therefore, in this embodiment of the invention, the most similar category corresponding to each category is identified, and then each sample corresponding to this category and the most similar category is identified. By screening these samples, it is determined whether category merging is required or whether the classification label of the sample needs to be changed.

[0096] As an optional implementation, step S304 above can be achieved by... Figure 8 Implemented as shown. Figure 8 A schematic flowchart of step S304 provided in an embodiment of the present invention:

[0097] S304-1, Identify the samples to be confirmed that have the same predicted classification from all samples with classification labels.

[0098] S304-2, determine the sum of disambiguation scores of all unconfirmed samples with the same predicted classification, and determine the predicted classification corresponding to the maximum sum of disambiguation scores as the most similar predicted classification;

[0099] S304-3, determine all unconfirmed samples corresponding to the most similar predicted classification as the first target samples, and determine the second target samples with the predicted classification label from all samples with the most similar predicted classification.

[0100] In this embodiment, since the predicted classification of each sample is obtained in advance, for each classification label, the sample predicted to be classified as that label can be determined as the second target sample from all samples. Furthermore, since each classification label corresponds to multiple samples, the sample predicted to be classified as the most similar to the predicted classification can be determined as the first target sample.

[0101] In this embodiment, the first target sample and the second target sample are used to instruct the implementation of a merging strategy on the classification label and the most similar predicted classification, or to instruct the implementation of a correction strategy on the first target sample and / or the second target sample.

[0102] For example, based on the above results, operators can find similarities between two categories of mispredicted data (i.e., the first target sample and the second target sample). If there is a significant conflict, they can consider merging the categories or adjusting the corresponding samples.

[0103] For example, if the first target sample is labeled as class A, but the classification model predicts that the first target sample is class B, this means that it is reasonable to classify the same text as either class A or class B. At this point, the operators can determine whether the first target sample needs to be merged into categories, or whether the first target text needs to have its classification label changed so that it can be completely distinguished from other categories. The same applies to the second target category.

[0104] For ease of understanding, let's assume one of the category labels is "Why can't membership benefits be applied?". There are 10 samples with this category label. 5 samples are predicted to be in category S1, 3 samples are predicted to be in category S2, and 2 samples are predicted to be in category S3. Since we have already obtained the disambiguation score for each sample, we can sum the disambiguation scores of all samples corresponding to each category to get the sum of the disambiguation scores for that category. That is, we can sum the disambiguation scores of the 5 samples corresponding to S1, the 3 samples corresponding to S2, and the 2 samples corresponding to S3. The final sum of the disambiguation scores for each predicted category can be understood as: the weight that needs to be disambiguated for that category. The category with the largest sum of disambiguation scores is determined as the most similar predicted category for "Why can't membership benefits be applied?".

[0105] Please see Figure 9 , Figure 9 This is a schematic diagram illustrating the category disambiguation result analysis provided in an embodiment of the present invention. It can be seen that, for each category, the embodiment of the present invention can identify the most similar category, and then, through the classification results of the samples, obtain samples that may conflict. Figure 9 The results shown are presented to operations personnel for knowledge disambiguation, data cleaning, and improved labeling accuracy.

[0106] Therefore, in order to facilitate analysis by operators, the above results can be displayed in the following manner after obtaining the most similar sample corresponding to each sample, the most similar category corresponding to each category, and the sum of the first target sample and the second target sample that may conflict with each other in the embodiments of the present invention:

[0107] The following information is displayed in descending order of the sum of disambiguation scores: the classification label, the number of samples corresponding to the classification label, the most similar predicted classification corresponding to the classification label, the probability of the first target sample and the most similar predicted classification corresponding to the first target sample, and the probability of the classification label corresponding to the second target sample.

[0108] The images are displayed in descending order of disambiguation scores, along with each sample, its corresponding classification label, predicted classification, disambiguation score, and most similar sample.

[0109] The training set processing method for classification provided in this embodiment of the invention can be executed in a hardware device or as a software module. When the training set processing method for classification is implemented as a software module, this embodiment of the invention also provides an apparatus for the training set processing method for classification. Please refer to [link to relevant documentation]. Figure 10 , Figure 10 This is a functional block diagram of a training set processing device for classification provided in an embodiment of the present invention. The training set processing device 400 for classification may include:

[0110] The acquisition module 410 is used to acquire a classification sample set; the classification sample set includes multiple samples and a classification label corresponding to each sample; one classification label corresponds to at least one sample;

[0111] The determination module 420 is used to determine each predicted category corresponding to each sample, the probability of each predicted category, and the disambiguation score corresponding to each sample; wherein the disambiguation score represents the error score of the classification label corresponding to the sample.

[0112] The determining module 420 is further configured to, for each sample, determine the most similar sample corresponding to each sample from each sample having the target prediction classification corresponding to the sample, wherein the probability of the target prediction classification is the highest; the probability of the target prediction classification is the highest.

[0113] The determining module 420 is further configured to determine the most similar predicted category corresponding to each of the classification labels based on the disambiguation score, and based on the most similar predicted category, determine a first target sample predicted to be classified as the most similar predicted category from all samples with the classification label, and determine a second target sample predicted to be classified as the classification label from all samples with the most similar predicted category.

[0114] Wherein, the first target sample and the second target sample are used to indicate the implementation of a merging strategy for the classification label and the most similar predicted classification, or to indicate the implementation of a correction strategy for the first target sample and / or the second target sample; the most similar sample is used to indicate the correction of the sample.

[0115] It is understandable that the acquisition module 410 and the determination module 420 can be executed collaboratively. Figure 3 Each step in the process is used to achieve the corresponding technical effect.

[0116] In an optional implementation, the determining module 420 is specifically used to: obtain each predicted classification of each sample and the probability of each predicted classification using the trained classification model; for each sample, determine the maximum misclassification probability and confusion probability corresponding to the sample based on the comparison result between the predicted classification and the classification label corresponding to the sample; and determine the maximum value between the maximum misclassification probability and the confusion probability as the disambiguation score corresponding to the sample.

[0117] In an optional implementation, the determining module 420 is specifically configured to: determine multiple sample subsets and a predicted sample set corresponding to each sample subset from the training set based on the sample identifier corresponding to each sample; wherein the union of each sample subset and the predicted sample set corresponding to the sample subset is the classification sample set; within a sample subset, the remainder obtained by dividing the sample identifier of each sample by the number of sample subsets is the same, and the remainder is different from the remainder corresponding to each predicted sample in the predicted sample set; sequentially train the classification model using each sample subset, and use the trained classification model to predict the predicted sample set corresponding to the sample subset, thereby obtaining the predicted classification of each predicted sample in the predicted sample set and the probability of each predicted classification.

[0118] In an optional implementation, the determining module 420 is specifically configured to: determine a first maximum probability and a second maximum probability in descending order of probability, and use the difference between the first maximum probability and the second maximum probability as the confusion probability; if the predicted classification corresponding to the first maximum probability is consistent with the classification label, then the second maximum probability is used as the maximum misclassification probability; if the predicted classification corresponding to the first maximum probability is inconsistent with the classification label, then the first maximum probability is used as the maximum misclassification probability.

[0119] In an optional implementation, the determining module 420 is specifically configured to: determine, from all samples having the classification label, samples to be confirmed that have the same predicted classification; determine the sum of disambiguation scores of all samples to be confirmed that have the same predicted classification, and determine the predicted classification corresponding to the maximum sum of disambiguation scores as the most similar predicted classification; determine all samples to be confirmed corresponding to the most similar predicted classification as the first target sample, and determine, from all samples having the most similar predicted classification, a second target sample whose predicted classification is the classification label.

[0120] In an optional implementation, the determining module 420 is specifically used for: if the target predicted classification is consistent with the classification label corresponding to the sample, then outputting that the most similar sample is empty; if the target predicted classification is inconsistent with the classification label corresponding to the sample, extracting all candidate samples with the target predicted classification from the classification sample set, calculating the similarity between all the candidate samples and the sample, and determining the candidate sample with the highest similarity as the most similar sample.

[0121] In an optional implementation, the device may further include a display module for sequentially displaying, in descending order of the sum of the disambiguation scores, the classification label, the number of samples corresponding to the classification label, the most similar predicted classification corresponding to the classification label, the probability of the first target sample and the most similar predicted classification corresponding to the first target sample, and the probability of the second target sample and the classification label corresponding to the second target sample; and sequentially displaying, in descending order of the disambiguation scores, each sample, the classification label corresponding to the sample, the predicted classification, the disambiguation score, and the most similar sample.

[0122] This invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the training set processing method for classification as described in any of the foregoing embodiments. The computer-readable storage medium may be, but is not limited to, various media capable of storing program code, such as a USB flash drive, portable hard drive, ROM, RAM, PROM, EPROM, EEPROM, magnetic disk, or optical disk.

[0123] It should be understood that the apparatus and methods disclosed in this invention can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0124] In addition, the functional modules in the various embodiments of the present invention can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0125] The above are merely preferred embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the scope of protection of the invention. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

Claims

1. A training set processing method for classification, characterized in that, The method includes: Obtain a classification sample set; the classification sample set contains multiple samples and a classification label corresponding to each sample; one classification label corresponds to at least one sample; the samples are corpora, text, or images; Using the trained classification model, each predicted category and the probability of each predicted category are obtained for each sample. For each sample, based on the comparison between the predicted category and the classification label corresponding to the sample, the maximum misclassification probability and the confusion probability corresponding to the sample are determined. The maximum value between the maximum misclassification probability and the confusion probability is determined as the disambiguation score corresponding to the sample. The disambiguation score represents the degree of error of the classification label corresponding to the sample. For each sample, from all samples with the target predicted classification corresponding to the sample, determine the most similar sample corresponding to each sample, where the probability of the target predicted classification is the highest; Based on the disambiguation score, the most similar predicted category corresponding to each category label is determined. Based on the most similar predicted category, a first target sample predicted to be classified as the most similar predicted category is determined from all samples with the category label, and a second target sample predicted to be classified as the category label is determined from all samples with the most similar predicted category. Wherein, the first target sample and the second target sample are used to indicate the implementation of a merging strategy for the classification label and the most similar predicted classification, or to indicate the implementation of a correction strategy for the first target sample and / or the second target sample; the most similar sample is used to indicate the correction of the sample.

2. The method according to claim 1, characterized in that, Using the trained classification model, the predicted categories for each sample and the probability of each predicted category are obtained, including: Based on the sample identifier corresponding to each sample, determine multiple sample subsets and a prediction sample set corresponding to each sample subset from the training set; Wherein, the union of each sample subset and the predicted sample set corresponding to the sample subset is the classification sample set; within a sample subset, the remainder obtained by dividing the sample identifier of each sample by the number of sample subsets is the same, and the remainder is different from the remainder corresponding to each predicted sample in the predicted sample set; The classification model is trained sequentially using each of the sample subsets, and the trained classification model is used to predict the prediction sample set corresponding to the sample subset, so as to obtain the prediction class of each prediction sample in the prediction sample set and the probability of each prediction class.

3. The method according to claim 1, characterized in that, For each sample, based on the comparison between the predicted classification and the classification label corresponding to the sample, the maximum misclassification probability and confusion probability corresponding to the sample are determined, including: The first maximum probability and the second maximum probability are determined in descending order of probability, and the difference between the first maximum probability and the second maximum probability is taken as the confusion probability. If the predicted classification corresponding to the first maximum probability is consistent with the classification label, then the second maximum probability is taken as the maximum misclassification probability; If the predicted classification corresponding to the first maximum probability is inconsistent with the classification label, then the first maximum probability is taken as the maximum misclassification probability.

4. The method according to claim 1, characterized in that, Based on the disambiguation score, the most similar predicted category corresponding to each category label is determined. Based on the most similar predicted category, a first target sample predicted to be classified as the most similar predicted category is determined from all samples with the category label, and a second target sample predicted to be classified as the category label is determined from all samples with the most similar predicted category. This includes: From all samples with the classification label, identify the samples to be confirmed that have the same predicted classification; The sum of disambiguation scores of all samples to be confirmed that have the same predicted classification is determined, and the predicted classification corresponding to the largest sum of disambiguation scores is determined as the most similar predicted classification. All the samples to be confirmed corresponding to the most similar predicted classification are determined as the first target samples, and from all the samples with the most similar predicted classification, the second target samples predicted to be classified as the classification label are determined.

5. The method according to claim 1, characterized in that, For each sample, from all samples having the target predicted classification corresponding to the sample, determine the most similar sample corresponding to each sample, including: If the predicted classification of the target is consistent with the classification label corresponding to the sample, then the output of the most similar sample is empty; If the target predicted classification is inconsistent with the classification label corresponding to the sample, all candidate samples with the target predicted classification are extracted from the classification sample set, the similarity between all candidate samples and the sample is calculated, and the candidate sample with the highest similarity is determined as the most similar sample.

6. The method according to claim 4, characterized in that, The method further includes: The classification labels, the number of samples corresponding to the classification labels, the most similar predicted classification corresponding to the classification labels, the probability of the first target sample and the most similar predicted classification corresponding to the first target sample, and the probability of the second target sample and the classification labels corresponding to the second target sample are displayed in descending order of the sum of the disambiguation scores. The following information is displayed sequentially, in descending order of disambiguation scores: each sample, its corresponding classification label, predicted classification, disambiguation score, and the most similar sample.

7. A training set processing apparatus for classification, characterized in that, include: The acquisition module is used to acquire the classification sample set; The classification sample set contains multiple samples and a classification label corresponding to each sample; Each of the classification labels corresponds to at least one of the samples; the samples are corpora, text, or images. The determination module is used to obtain each predicted category and the probability of each predicted category for each sample using the trained classification model; for each sample, based on the comparison result between the predicted category and the classification label corresponding to the sample, the maximum misclassification probability and the confusion probability corresponding to the sample are determined; the maximum value between the maximum misclassification probability and the confusion probability is determined as the disambiguation score corresponding to the sample; wherein the disambiguation score represents the error score of the classification label corresponding to the sample. The determining module is further configured to, for each sample, determine the most similar sample corresponding to each sample from among the samples having the target prediction classification corresponding to the sample, wherein the probability of the target prediction classification is the highest; the probability of the target prediction classification is the highest. The determining module is further configured to determine the most similar predicted category corresponding to each category label based on the disambiguation score, and based on the most similar predicted category, determine a first target sample predicted to be classified as the most similar predicted category from all samples with the category label, and determine a second target sample predicted to be classified as the category label from all samples with the most similar predicted category. Wherein, the first target sample and the second target sample are used to indicate the implementation of a merging strategy for the classification label and the most similar predicted classification, or to indicate the implementation of a correction strategy for the first target sample and / or the second target sample; the most similar sample is used to indicate the correction of the sample.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program executable by the processor, the processor being able to execute the computer program to implement the method of any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 6.

Images