Classification accuracy evaluation method, device, computer equipment and storage medium

Abstract

The present application relates to a classification accuracy evaluation method, device, computer equipment and storage medium. The method comprises: obtaining a harmful speech sample set; inputting each harmful speech sample in the harmful speech sample set into a harmful speech classification model to be evaluated for classification, and obtaining a predicted category label; determining a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample in a preset classification hierarchy; and calculating the classification accuracy of the harmful speech classification model to be evaluated according to the target classification. In this scheme, the harmful speech samples are classified at multiple levels (i.e., classification hierarchy), and then the target classification to which the predicted category label and the sample category label belong is determined in the classification hierarchy. The target classification can reflect the matching degree between the predicted category label and the sample category label, and then the classification accuracy of the classification model is determined according to the target classification, which can effectively improve the accuracy of the classification model evaluation.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to a classification accuracy evaluation method, apparatus, computer equipment and storage medium. Background Art

[0002] Classification technology is a new and very effective machine learning method developed on the basis of statistical learning theory. Due to its excellent learning performance, classification technology has become a new research hotspot in the prediction field.

[0003] The application of speech classification has penetrated into all aspects of daily life. For example, harmful speech classification has achieved remarkable results in preventing various telecommunications frauds. The harmful speech classification model is a model used to classify harmful speech. The classification results of harmful speech can in turn evaluate the classification effect of the harmful speech classification model.

[0004] At present, the evaluation of harmful speech classification models is based on whether the correct results are consistent with the predicted results to evaluate the classification effect of the model. It is unable to make a more refined evaluation of the specific classification results, and the accuracy of the evaluation of the classification model is low. Summary of the invention

[0005] Based on this, it is necessary to provide a classification accuracy evaluation method, device, computer equipment and storage medium that can improve the accuracy of classification model evaluation in response to the above technical problems.

[0006] In a first aspect, a classification accuracy evaluation method is provided, the method comprising:

[0007] Obtaining a harmful speech sample set, the harmful speech sample set including a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample;

[0008] Input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtain a predicted category label;

[0009] In a preset classification hierarchy, determining a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample; wherein the target classification is a classification of the lowest level including the predicted category label and the sample category label;

[0010] The classification accuracy of the harmful speech classification model to be evaluated is determined according to the target classification.

[0011] In one embodiment, determining a target classification corresponding to a predicted class label and a sample class label of a harmful speech sample in a preset classification hierarchy includes:

[0012] In the preset classification hierarchy, searching upward layer by layer and comparing the upper classification of the predicted category label and the upper classification of the sample category label of the harmful speech sample;

[0013] When the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is taken as the target classification.

[0014] In one embodiment, determining the classification accuracy of the harmful speech classification model to be evaluated according to the target classification includes:

[0015] Determine the parasitic minimum class group of each sample label according to the target classification and the minimum class not in the target classification, wherein the minimum class is the classification corresponding to each sample class label in the harmful speech sample set;

[0016] Calculate the classification cutoff according to the parasitic minimum group corresponding to each sample;

[0017] Add up the classification cuts of all samples to get the total classification cut;

[0018] The classification accuracy is calculated based on the total classification cut and the total maximum classification cut.

[0019] In one embodiment, calculating the classification cutoff according to the parasitic minimum cluster corresponding to each sample includes:

[0020] When the predicted category label of the harmful speech sample is the same as the sample category label, the classification cutoff is calculated based on the number of samples in each smallest class in the set of smallest classes and the total number of samples;

[0021] When the predicted category label of the harmful speech sample is different from the sample category label, the classification cutoff is calculated based on the number of samples in the target category, the number of samples of each smallest class that is not in the target category, and the total number of samples.

[0022] In one embodiment, the classification accuracy includes the total classification cut rate; the classification accuracy is calculated based on the total classification cut amount and the total maximum classification cut amount, including:

[0023] Calculate the maximum classification cut according to the minimum category of the sample category label, and calculate the total maximum classification cut according to the maximum classification cut;

[0024] The total classification cut rate is calculated based on the total classification cut amount and the total maximum classification cut amount.

[0025] In one embodiment, the classification accuracy includes a total classification loss rate;

[0026] The classification accuracy is calculated based on the total classification cut and the total maximum classification cut, including:

[0027] Calculate the total classified loss amount based on the total classified cut amount and the total maximum classified cut amount;

[0028] The total classified loss rate is calculated based on the total classified loss amount and the total maximum classified cut amount.

[0029] In one embodiment, the process of determining the classification level includes:

[0030] The classification corresponding to each sample category label in the harmful speech sample set is taken as the smallest class of the classification level;

[0031] Performing classification processing according to the smallest class and a preset iterative classification algorithm to obtain at least one classification set, wherein a classification set includes the categories obtained by one classification;

[0032] A classification hierarchy is established according to the association relationship between the smallest class and the classifications included in at least one classification set.

[0033] In a second aspect, a classification accuracy evaluation device is provided, the device comprising:

[0034] An acquisition module is used to acquire a harmful speech sample set, where the harmful speech sample set includes a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample;

[0035] A classification module, used for inputting each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtaining a predicted category label;

[0036] A target classification determination module determines a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample in a preset classification hierarchy; wherein the target classification is a classification of the lowest level including the predicted category label and the sample category label;

[0037] The determination module is used to determine the classification accuracy of the harmful speech classification model to be evaluated according to the target classification.

[0038] In a third aspect, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0039] Obtaining a harmful speech sample set, the harmful speech sample set including a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample;

[0040] Input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtain a predicted category label;

[0041] In a preset classification hierarchy, determining a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample; wherein the target classification is a classification of the lowest level including the predicted category label and the sample category label;

[0042] The classification accuracy of the harmful speech classification model to be evaluated is determined according to the target classification.

[0043] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Obtaining a harmful speech sample set, the harmful speech sample set including a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample;

[0045] Input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtain a predicted category label;

[0046] In a preset classification hierarchy, determining a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample; wherein the target classification is a classification of the lowest level including the predicted category label and the sample category label;

[0047] The classification accuracy of the harmful speech classification model to be evaluated is determined according to the target classification.

[0048] The above classification accuracy evaluation method, device, computer equipment and storage medium obtain a harmful speech sample set, which includes multiple harmful speech samples and a sample category label corresponding to each harmful speech sample; input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtain a predicted category label; determine the target classification corresponding to the predicted category label and the sample category label of the harmful speech sample in the preset classification hierarchy; wherein the target classification is the lowest level classification including the predicted category label and the sample category label; and determine the classification accuracy of the harmful speech classification model to be evaluated according to the target classification. In this scheme, the harmful speech samples are classified at multiple levels (i.e., classification hierarchy), and then the target classification to which the predicted category label and the sample category label belong together is determined in the classification hierarchy. The target classification can reflect the matching degree of the predicted category label and the sample category label, and then the classification accuracy of the classification model is determined according to the target classification. Compared with the scheme in the prior art that directly compares whether the predicted category label and the sample category label are consistent to evaluate the classification effect of the model, the accuracy of the classification model evaluation can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of a flow chart of a classification accuracy evaluation method in one embodiment;

[0050] Figure 2 is a schematic diagram of a classification hierarchy in one embodiment;

[0051] Figure 3 is a structural block diagram of a classification accuracy evaluation device in one embodiment;

[0052] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0054] When classifying harmful speech, it is very important to evaluate the harmful speech classification model through the classification results of harmful speech. The evaluation mainly evaluates the quality of a certain aspect of the model by calculating the corresponding indicators, so that different models have common comparable indicators. Furthermore, these indicators can also be used as an optimization target to optimize the model.

[0055] Currently, common evaluation criteria for harmful speech classification models include accuracy, error rate, sensitivity, precision, recall rate, Kappa coefficient, etc. However, these classification evaluation criteria cannot effectively utilize the prediction results of the model. For the prediction results of the model, only the information of whether it is a correct result or an incorrect result can be used, but the information of how close it is to the correct result cannot be used.

[0056] For example, in the classification of harmful speech, the sample classification labels include A1, A2, B1, and B2, where A1, A2, B1, and B2 represent different categories. A1 refers to consumer fraud speech, A2 refers to impersonation fraud speech, B1 refers to hard violence speech, and B2 refers to soft violence speech. A1 and A2 belong to the same category A, i.e., fraud speech, and B1 and B2 belong to the same category B, i.e., violent speech.

[0057] Suppose there are three harmful speech samples, namely sample 1, sample 2, and sample 3, and the sample classification labels of these three samples are A1, B1, and A2. There are two harmful speech classification models to be evaluated, namely model 1 and model 2. The classification results of harmful speech samples by model 1 are A1, B2, and A2, and the classification results of harmful speech samples by model 2 are A1, A2, and A2. Although both classification results are correct for 2 and wrong for 1, the evaluation results of the existing classification model evaluation indicators such as accuracy, error rate, and recall rate are consistent. However, it is obvious that the classification result of harmful speech sample 2 by model 1 is closer to the correct result. Both belong to the A category. Although it cannot be accurately classified into each subcategory, all the major categories are judged correctly, while model 2 even misjudges the harmful speech type as fraud or violence.

[0058] Therefore, although neither Model 1 nor Model 2 can accurately judge and classify harmful speech samples, Model 1 has better "classification and narrowing" capabilities than Model 2 and can provide more information in samples. The existing classification evaluation indicators are not precise enough for model evaluation, do not fully utilize the information of classification results, and the accuracy of the evaluation of harmful speech classification models is not high.

[0059] Therefore, the classification accuracy evaluation method provided by the present invention makes full use of the classification results of the harmful speech classification model, improves the precision of the evaluation of the classification results, and enables this classification method to reflect the amount of information brought by the erroneous classification results, thereby improving the accuracy of the evaluation of the harmful speech classification model.

[0060] In one embodiment, Figure 1 As shown, a classification accuracy evaluation method is provided. This embodiment takes the application of this method to a terminal as an example. It is understandable that this method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The classification accuracy evaluation method provided in this application can not only be used to evaluate harmful speech classification models, but can also be applied to other fields to evaluate other classification models, for example, to evaluate medical text classification models and song classification models. This embodiment of the application takes the evaluation of harmful speech classification models as an example, and does not limit the evaluation of other classification models. In this embodiment, the method includes the following steps:

[0061] Step 102: Obtain a harmful speech sample set.

[0062] The harmful speech sample set includes multiple harmful speech samples and a sample category label corresponding to each harmful speech sample. The harmful speech sample set is a data set used to test the classification effect of the harmful speech classification model. The harmful speech sample set includes multiple harmful speech samples with labels (i.e., sample category labels). The sample category label is the correct classification label of the harmful speech sample. The sample category label is a label corresponding to the harmful speech sample that is artificially set in advance. The sample category labels of some harmful speech samples in the harmful speech sample set are the same, and the sample category labels of the remaining harmful speech samples are different.

[0063] Specifically, the terminal obtains the harmful sample data set from a database or a memory.

[0064] Step 104: input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtain a predicted category label.

[0065] Specifically, the terminal inputs each harmful speech sample in the acquired harmful speech sample set into the harmful speech classification model to be evaluated, and classifies each harmful speech sample through the harmful speech classification model to obtain a predicted category label corresponding to each harmful speech sample. The harmful speech classification model can be obtained by training according to a variety of machine learning algorithms or neural network models, which is not limited in the embodiments of the present application.

[0066] Step 106: Determine a target category corresponding to the predicted category label and the sample category label of the harmful speech sample in a preset classification hierarchy.

[0067] Among them, the target classification is the lowest level classification that contains the predicted category label and the sample category label.

[0068] Specifically, the terminal may pre-store a classification hierarchy, which includes multiple layers. In order from low to high, the number of categories contained in each layer decreases, and the association relationship between the categories of multiple harmful speech samples in the harmful speech sample set at different levels. Figure 2 As shown in the figure, {a11, a12, a13, a21, b11, b21, b22} are the 7 smallest classes corresponding to the sample category labels of multiple harmful speech samples. These 7 smallest classes constitute the bottom layer of the classification hierarchy. The 7 smallest classes in the smallest class group are classified according to similarity to obtain four parent classes {a1, a2, b1, b2}. The four parent classes are classified to obtain two ancestor classes {A, B}. The last two ancestor classes belong to the general class. The specific generation process of the classification hierarchy will be described in detail later.

[0069] The terminal can determine the level and category at which the harmful speech sample is correctly classified by searching upward layer by layer in the preset classification hierarchy for the category in the lowest classification hierarchy to which the sample category label of the harmful speech sample and the predicted category label obtained by classification belong together.

[0070] For example, the predicted category label of sample 1 is a11, while the sample category label of sample 1 is a21, and the lowest classification level to which a11 and a21 belong together is A, so the target classification is A. The predicted category label of sample 2 is a13, while the sample category label of sample 2 is b12, and the lowest classification level to which a13 and b12 belong together is the general category, so the target classification is the general category. Obviously, the harmful speech classification model can classify sample 1 to a lower correct level, and the classification accuracy is higher than that of sample 2.

[0071] Step 108: Determine the classification accuracy of the harmful speech classification model to be evaluated according to the target classification.

[0072] Specifically, the classification accuracy of the harmful speech classification model to be evaluated is calculated according to the levels at which multiple harmful speech samples in the harmful speech sample set are correctly classified and the classification.

[0073] In the above classification accuracy evaluation method, a harmful speech sample set is obtained, the harmful speech sample set includes multiple harmful speech samples and a sample category label corresponding to each harmful speech sample; each harmful speech sample in the harmful speech sample set is input into the harmful speech classification model to be evaluated for classification to obtain a predicted category label; in a preset classification hierarchy, a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample is determined; wherein the target classification is the lowest level classification including the predicted category label and the sample category label; and the classification accuracy of the harmful speech classification model to be evaluated is determined according to the target classification. In this scheme, the harmful speech samples are classified at multiple levels (i.e., classification levels), and then the target classification to which the predicted category label and the sample category label belong together is determined in the classification level. The target classification can reflect the matching degree of the predicted category label and the sample category label, and then the classification accuracy of the classification model is determined according to the target classification. Compared with the scheme in the prior art that directly compares whether the predicted category label and the sample category label are consistent to evaluate the classification effect of the model, the accuracy of the classification model evaluation can be effectively improved.

[0074] In an optional embodiment, the process of determining the classification level includes: taking the classification corresponding to each sample category label in the harmful speech sample set as the minimum class of the classification level; performing classification processing according to the minimum class and a preset iterative classification algorithm to obtain at least one classification set, wherein a classification set contains the classifications obtained by one classification; and establishing the classification level according to the association relationship between the minimum class and the classifications contained in at least one classification set.

[0075] The classification process refers to clustering the smallest class or classification set and giving specific attribute names to the clustering results.

[0076] Specifically, the classification corresponding to the sample category labels of multiple harmful speech samples in the harmful speech sample set is taken as the minimum class, the set of multiple minimum classes is taken as the minimum class group, and the minimum class group is taken as the bottom layer of the classification hierarchy. The minimum classes are classified according to the similarities between the multiple minimum classes in the minimum class group, at least one minimum class is classified as a classification (parent class), and an association relationship is established between the parent class and at least one minimum class. The parent class of each minimum class is taken as a classification set, and the classification set is taken as the upper layer of the bottom layer. Then, the parent class of each minimum class is classified again, at least one parent class is classified as an ancestor class, and an association relationship is established between the ancestor class and at least one parent class. And so on, until the total class is obtained, and the final classification result is the total class. Based on the above process, the classification hierarchy can be obtained.

[0077] The preset iterative classification algorithm can classify the sample set into different levels according to the classification habits or common sense corresponding to different types of data sets. For example, the "harmful speech" category includes the "fraud" category and the "violence" category, the "fraud" category includes the "consumer fraud" category and the "impersonation fraud" category, and the "violence" category includes the "hard violence" category and the "soft violence" category.

[0078] For sample sets that do not have corresponding hierarchical structures in daily life, the preset iterative classification algorithm can also use a clustering algorithm. By setting the number of clustering categories, the number of the previous layer is α times the number of the next layer. α is determined by the fineness of the data set categories and is usually taken as 0.5.

[0079] like Figure 2 In the classification hierarchy shown, there are 10 harmful speech samples in the harmful speech sample set. These 10 samples have 7 sample category labels, namely a11, a12, a13, a14, b11, b21, and b22. These 7 sample category labels can divide the 10 harmful speech samples into 7 smallest classes. The set of 7 smallest classes is the smallest class group {a11, a12, a13, a21, b11, b21, b22}, and the smallest class group is the bottom layer of the classification hierarchy. The 7 smallest classes in the smallest class group are classified according to similarity to obtain four parent classes {a1, a2, b1, b2}. The four parent classes are classified to obtain two ancestor classes {A, B}, and finally the whole is classified into the general class.

[0080] In this embodiment, by establishing a classification hierarchy corresponding to the harmful speech sample set, the more refined the classification level in the classification hierarchy, the lower the level corresponding to the level, and the more classification levels there are, the more refined the classification is, so that more information can be mined from the harmful speech samples. Therefore, the classification results of the harmful speech model can be evaluated more finely according to the classification hierarchy, making the evaluation results more accurate.

[0081] In an optional embodiment, in a preset classification hierarchy, a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample is determined, including: in the preset classification hierarchy, searching upward layer by layer and comparing the upper-level classification of the predicted category label and the upper-level classification of the sample category label of the harmful speech sample; when the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is used as the target classification.

[0082] Specifically, if the predicted category label is the same as the sample category label of the harmful speech sample, the prediction is correct, and the target classification is the smallest class corresponding to the sample category label in the preset classification hierarchy.

[0083] If the predicted category label is different from the sample category label of the harmful speech sample, then in the preset classification hierarchy, starting from the bottom layer, search and compare the upper-level classification of the predicted category label and the upper-level classification of the sample category label of the harmful speech sample layer by layer. When the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is used as the target classification.

[0084] For example, for a certain sample, the sample category label is a11, and the predicted category label classified by the harmful speech classification model is a12. The lowest upper classification of the sample category label a11 and the predicted category label a12 is a1, so the lowest level that the harmful speech classification model can correctly classify the sample is the level where a1 is located, because the classification below a1 is wrong, so the target classification is a1.

[0085] In an optional embodiment, the classification accuracy of the harmful speech classification model to be evaluated is calculated according to the target classification, including: determining the parasitic minimum class group of each sample label according to the target classification and the minimum class not in the target classification, wherein the minimum class is the classification corresponding to each sample category label in the harmful speech sample set; calculating the classification cut according to the parasitic minimum class group corresponding to each sample; adding the classification cuts of all samples to obtain the total classification cut; and calculating the classification accuracy according to the total classification cut and the total maximum classification cut.

[0086] Specifically, for each harmful speech sample, a set consisting of a target classification and a minimum class that is not in the target classification is taken as the parasitic minimum class group of the harmful speech sample. The classification cut is calculated based on the number of samples of the target classification, the number of samples of the minimum class, and the total number of samples in the parasitic minimum class group. The classification cut of each sample in the harmful speech sample set is added together to obtain a total classification cut. The maximum classification cut is calculated based on the correct classification result of the sample by the harmful speech classification model, and then the total maximum classification cut is calculated based on the number of samples and the maximum classification cut. The classification accuracy is calculated based on the total classification cut obtained by the predicted category label and the total maximum classification cut obtained by the correct classification.

[0087] In an optional embodiment, a classification cut is calculated based on the parasitic minimum class group corresponding to each sample, including: when the predicted category label of the harmful speech sample is the same as the sample category label, the classification cut is calculated based on the number of samples in each minimum class in the set of minimum classes and the total number of samples; when the predicted category label of the harmful speech sample is different from the sample category label, the classification cut is calculated based on the number of samples in the target classification, the number of samples in each minimum class that is not in the target classification, and the total number of samples.

[0088] Specifically, for each harmful speech sample, the classification cutoff Q represents the maximum classification strength for the sample. The classification cutoffs of different harmful speech samples are added together to obtain the total classification cutoff Q of the harmful speech sample set. all .

[0089] Among them, y i represents the i-th smallest class, p i Represents y i The proportion of samples in the smallest class represented by the n i Refers to y i represents the number of samples in the smallest class, N represents the total number of samples in the harmful speech sample set, U represents the target classification, and p j It indicates the proportion of samples in the target category to the total number.

[0090] When the predicted category label of the harmful speech sample is the same as the sample category label, the parasitic minimum class group corresponding to the harmful speech sample is the set of minimum classes in the preset classification hierarchy. The classification cutoff is calculated based on the number of samples in each minimum class in the set of minimum classes and the total number of samples.

[0091] When the predicted category label of the harmful speech sample is different from the sample category label, the parasitic minimum class group corresponding to the harmful speech sample is the target category and the minimum class that is not in the target category. The classification cutoff is calculated based on the number of samples in the target category, the number of samples in each minimum class that is not in the target category, and the total number of samples.

[0092] In an optional embodiment, the classification accuracy includes the total classification cut rate; accordingly, the classification accuracy is calculated based on the total classification cut amount and the total maximum classification cut amount, including: calculating the maximum classification cut amount based on the minimum class of the sample category label, calculating the total maximum classification cut amount based on the maximum classification cut amount; calculating the total classification cut rate based on the total classification cut amount and the total maximum classification cut amount.

[0093] Specifically, when a harmful speech sample is correctly classified into the smallest class, the corresponding classification cutoff is the maximum classification cutoff. For a data set containing N harmful speech samples, the total maximum classification cutoff Q allmax =N×Q max .

[0094] The total classification cutting rate R is calculated based on the ratio of the total classification cutting amount to the total maximum classification cutting amount. cut ,

[0095] In an optional embodiment, the classification accuracy includes the total classification loss rate; accordingly, the classification accuracy is calculated based on the total classification amount and the total maximum classification amount, including: calculating the total classification loss amount based on the total classification amount and the total maximum classification amount; calculating the total classification loss rate based on the total classification loss amount and the total maximum classification amount.

[0096] Specifically, the total classification loss L is calculated based on the difference between the total classification cutting amount and the total maximum classification cutting amount, L = Q allmax -Q all The total classification loss rate R is calculated based on the ratio of the total classification loss to the total maximum classification cut. loss ,

[0097] In this embodiment, by accurately representing the level to which the classification results can be correctly classified based on the total classification loss rate and the total classification cutoff, the harmful speech classification model can be evaluated more accurately, and the model can be further optimized based on the classification accuracy, thereby improving the prediction effect of the model.

[0098] For example, the classification effect of the harmful speech classification model can be evaluated by any one of the total classification loss rate and the total classification cutoff. When the classification effect calculated by any one of the indicators is greater than a manually set threshold, a trained harmful speech classification model is obtained. The classification effect of the harmful speech classification model can also be evaluated by the total classification loss rate and the total classification cutoff. When the classification effect calculated by the two indicators is greater than a manually set threshold, a trained harmful speech classification model is obtained.

[0099] In order to facilitate the understanding of the technical solution provided in the embodiment of the present application, the classification accuracy evaluation method provided in the embodiment of the present application is briefly described with a complete classification accuracy evaluation process:

[0100] (1) Obtain a harmful speech sample set, where the harmful speech sample set includes multiple harmful speech samples and a sample category label corresponding to each harmful speech sample.

[0101] (2) Each harmful speech sample in the harmful speech sample set is input into the harmful speech classification model to be evaluated for classification to obtain a predicted category label.

[0102] (3) In the preset classification hierarchy, searching upward layer by layer and comparing the upper classification of the predicted category label and the upper classification of the sample category label of the harmful speech sample;

[0103] When the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is taken as the target classification.

[0104] (4) determining the parasitic minimum class group of each sample label according to the target classification and the minimum class not in the target classification, wherein the minimum class is the classification corresponding to each sample class label in the harmful speech sample set;

[0105] Calculate the classification cutoff according to the parasitic minimum group corresponding to each sample;

[0106] Add up the classification cuts of all samples to get the total classification cut;

[0107] Calculate the maximum classification cut according to the minimum category of the sample category label, and calculate the total maximum classification cut according to the maximum classification cut;

[0108] The total classification cut rate is calculated based on the total classification cut amount and the total maximum classification cut amount.

[0109] (5) Calculate the total classified loss amount based on the total classified cut amount and the total maximum classified cut amount;

[0110] The total classified loss rate is calculated based on the total classified loss amount and the total maximum classified cut amount.

[0111] like Figure 2As shown, the following takes the classification accuracy evaluation method provided in the embodiment of the present application to evaluate the classification results of harmful speech classification model 1 and harmful speech classification model 2 as an example to illustrate the detailed classification accuracy evaluation process:

[0112] (1) Obtain a harmful speech sample set. There are 10 harmful speech samples in the harmful speech sample set. These 10 samples have 7 sample category labels, namely a11, a12, a13, a14, b11, b21, and b22. These 7 sample category labels can divide the 10 harmful speech samples into 7 smallest classes. The set of 7 smallest classes is the smallest class group {a11, a12, a13, a21, b11, b21, b22}. The smallest class group is the bottom layer of the classification hierarchy. The 7 smallest classes in the smallest class group are classified according to similarity to obtain four parent classes {a1, a2, b1, b2}. The four parent classes are classified to obtain two ancestor classes {A, B}, and finally the whole is classified into the general class. A stands for "fraud" and B stands for "violence"; a1 stands for "impersonation fraud", a2 stands for "consumer fraud", b1 stands for "hard violence" and b2 stands for "soft violence"; a11 stands for "impersonating an acquaintance", a12 stands for "impersonating a bank", a13 stands for "impersonating the police", a21 stands for "selling counterfeit goods", b11 stands for "containing violent words", b21 stands for "insults and abuse", and b22 stands for "disclosing other people's privacy".

[0113] The sample category labels of the 10 harmful speech samples are: {a13, b21, a11, a12, b11, b22, a21, a12, b22, a21}.

[0114] (2) The 10 harmful speech samples were classified using harmful speech classification model 1 and harmful speech classification model 2. The classification results of model 1 were {a13, b22, a11, a12, b11, b22, a21, a21, b22, a21}, of which only the second and eighth samples were classified incorrectly; the classification results of model 2 were {a12, b21, a11, a12, b11, b21, a21, a12, b22, a21}, of which only the first and sixth samples were classified incorrectly.

[0115] (3) In the preset classification hierarchy, find the target classification of the classification result and the correct sample category label. For each sample, the target classification corresponding to each sample in the classification result of model 1 is {a13, b2, a11, a12, b11, b22, a21, A, b22, a21}, and the target classification corresponding to each sample in the classification result of model 2 is {a1, b21, a11, a12, b11, b2, a21, a12, b22, a21}.

[0116] (4) The smallest class is {a11,a12,a13,a21,b11,b21,b22}, and the proportion of their respective samples in the total population is: {0.1,0.2,0.1,0.2,0.1,0.1,0.2}.

[0117] For each sample classified by model 1, find its minimum parasitic cluster:

[0118] The first sample is classified correctly, and its smallest parasitic cluster is {a11, a12, a13, a21, b11, b21, b22};

[0119] The second sample is misclassified, and its target classification is d, so its parasitic minimum cluster is {a11, a12, a13, a21, b11, b2};

[0120] The third sample is correctly classified, and its smallest parasitic cluster is {a11,a12,a13,a21,b11,b21,b22};

[0121] …

[0122] The eighth sample is misclassified, and its minimal class is A, so its parasitic minimal class group is {A, b11, b21, b22};

[0123] …

[0124] The 10th sample is correctly classified, and its parasitic minimum cluster is {a11, a12, a13, a21, b11, b21, b22};

[0125] For each sample's classification result, the classification cutoff is calculated based on its parasitic minimum cluster.

[0126] First calculate the maximum classification cut

[0127] Q allmax =N×Q max =27.22,

[0128] It is easy to see that for correctly classified samples, the classification cut is equal to the maximum classification cut. The classification cutoff of the parasitic minimum group obtained by model 1 to classify the i-th sample is

[0129]

[0130]

[0131]

[0132] but

[0133] Total classification rate

[0134] Total classified losses

[0135] Total classified loss rate

[0136] Similarly, the total classification rate of model 2 can be calculated

[0137] Total classified loss rate

[0138] Whether from the perspective of total classified loss amount or total classified loss rate, Model 2 has better classification effect.

[0139] If the traditional classification evaluation method is used to evaluate Model 1 and Model 2, the accuracy of the two models is 80%. The traditional classification model evaluation standard cannot distinguish the classification accuracy of the two models.

[0140] At the same time, the total classification loss or the total classification loss rate can be used as the loss function in the machine learning model. The model can be optimized with the goal of reducing the total classification loss and the total classification loss rate, which can be effective in areas such as harmful speech classification algorithm evaluation.

[0141] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0142] In one embodiment, Figure 3 As shown, a classification accuracy evaluation device is provided, including: an acquisition module 302, a classification module 304, a target classification determination module 306 and a determination module 308, wherein:

[0143] The acquisition module 302 is used to acquire a harmful speech sample set, where the harmful speech sample set includes a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample.

[0144] The classification module 304 is used to input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, so as to obtain a predicted category label.

[0145] The target classification determination module 306 determines the target classification corresponding to the predicted category label and the sample category label of the harmful speech sample in a preset classification hierarchy; wherein the target classification is the lowest level classification including the predicted category label and the sample category label.

[0146] The determination module 308 is used to determine the classification accuracy of the harmful speech classification model to be evaluated according to the target classification.

[0147] In an optional embodiment, the target classification determination module 306 is also used to search and compare the upper-level classification of the predicted category label and the upper-level classification of the sample category label of the harmful speech sample in a preset classification hierarchy layer by layer; and when the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is used as the target classification.

[0148] In an optional embodiment, the determination module 308 is also used to determine the parasitic minimum class group of each sample label based on the target classification and the minimum class that is not in the target classification, wherein the minimum class is the classification corresponding to each sample category label in the harmful speech sample set; and calculate the classification cut according to the parasitic minimum class group corresponding to each sample; and add the classification cuts of all samples to obtain the total classification cut; and calculate the classification accuracy based on the total classification cut and the total maximum classification cut.

[0149] In an optional embodiment, the determination module 308 is also used to calculate the classification cut according to the number of samples in each smallest class in the set of smallest classes and the total number of samples when the predicted category label of the harmful speech sample is the same as the sample category label; and when the predicted category label of the harmful speech sample is different from the sample category label, calculate the classification cut according to the number of samples in the target classification, the number of samples in each smallest class that is not in the target classification, and the total number of samples.

[0150] In an optional embodiment, the classification accuracy includes the total classification cut rate; the determination module 308 is also used to calculate the maximum classification cut amount based on the minimum class of the sample category label, calculate the total maximum classification cut amount based on the maximum classification cut amount; and calculate the total classification cut rate based on the total classification cut amount and the total maximum classification cut amount.

[0151] In an optional embodiment, the classification accuracy includes the total classification loss rate; the determination module 308 is also used to calculate the total classification loss amount based on the total classification amount and the total maximum classification amount; and calculate the total classification loss rate based on the total classification loss amount and the total maximum classification amount.

[0152] In an optional embodiment, the classification accuracy evaluation device also includes a classification hierarchy establishment module, which is used to take the classification corresponding to each sample category label in the harmful speech sample set as the minimum class of the classification hierarchy; and perform classification processing according to the minimum class and a preset iterative classification algorithm to obtain at least one classification set, wherein a classification set contains the classification obtained by one classification; and establish a classification hierarchy according to the association relationship between the minimum class and the classification contained in at least one classification set.

[0153] For the specific definition of the classification accuracy evaluation device, please refer to the definition of the classification accuracy evaluation method above, which will not be repeated here. Each module in the above classification accuracy evaluation device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0154] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a classification accuracy evaluation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0155] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0156] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0157] Obtaining a harmful speech sample set, the harmful speech sample set including a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample;

[0158] Input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtain a predicted category label;

[0159] In a preset classification hierarchy, determining a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample; wherein the target classification is a classification of the lowest level including the predicted category label and the sample category label;

[0160] The classification accuracy of the harmful speech classification model to be evaluated is determined according to the target classification.

[0161] In one embodiment, when the processor executes the computer program, the following steps are also implemented: in a preset classification hierarchy, a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample is determined, including: in the preset classification hierarchy, searching upward layer by layer and comparing the upper-level classification of the predicted category label and the upper-level classification of the sample category label of the harmful speech sample; when the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is used as the target classification.

[0162] In one embodiment, when the processor executes the computer program, the following steps are also implemented: determining the classification accuracy of the harmful speech classification model to be evaluated according to the target classification, including: determining the parasitic minimum class group of each sample label according to the target classification and the minimum class that is not in the target classification, wherein the minimum class is the classification corresponding to each sample category label in the harmful speech sample set; calculating the classification cut according to the parasitic minimum class group corresponding to each sample; adding the classification cuts of all samples to obtain the total classification cut; calculating the classification accuracy according to the total classification cut and the total maximum classification cut.

[0163] In one embodiment, when the processor executes the computer program, the following steps are also implemented: calculating the classification cut according to the parasitic minimum class group corresponding to each sample, including: when the predicted category label of the harmful speech sample is the same as the sample category label, calculating the classification cut according to the number of samples in each minimum class in the set of minimum classes and the total number of samples; when the predicted category label of the harmful speech sample is different from the sample category label, calculating the classification cut according to the number of samples in the target classification, the number of samples in each minimum class that is not in the target classification, and the total number of samples.

[0164] In one embodiment, the processor also implements the following steps when executing the computer program: the classification accuracy includes the total classification cut rate; the classification accuracy is calculated based on the total classification cut amount and the total maximum classification cut amount, including: calculating the maximum classification cut amount based on the minimum class of the sample category label, and calculating the total maximum classification cut amount based on the maximum classification cut amount; calculating the total classification cut rate based on the total classification cut amount and the total maximum classification cut amount.

[0165] In one embodiment, the processor also implements the following steps when executing the computer program: the classification accuracy includes the total classification loss rate; the classification accuracy is calculated based on the total classification cut amount and the total maximum classification cut amount, including: calculating the total classification loss amount based on the total classification cut amount and the total maximum classification cut amount; calculating the total classification loss rate based on the total classification loss amount and the total maximum classification cut amount.

[0166] In one embodiment, the processor further implements the following steps when executing the computer program: a process for determining a classification level, including: taking the classification corresponding to each sample category label in the harmful speech sample set as the minimum class of the classification level; performing classification processing according to the minimum class and a preset iterative classification algorithm to obtain at least one classification set, wherein a classification set contains the classification obtained by one classification; and establishing a classification level according to the association relationship between the minimum class and the classification contained in at least one classification set.

[0167] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0168] Obtaining a harmful speech sample set, the harmful speech sample set including a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample;

[0169] Input each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtain a predicted category label;

[0170] In a preset classification hierarchy, determining a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample; wherein the target classification is a classification of the lowest level including the predicted category label and the sample category label;

[0171] The classification accuracy of the harmful speech classification model to be evaluated is determined according to the target classification.

[0172] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: in a preset classification hierarchy, a target classification corresponding to the predicted category label and the sample category label of the harmful speech sample is determined, including: in the preset classification hierarchy, searching upward layer by layer and comparing the upper-level classification of the predicted category label and the upper-level classification of the sample category label of the harmful speech sample; when the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is used as the target classification.

[0173] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: determining the classification accuracy of the harmful speech classification model to be evaluated according to the target classification, including: determining the parasitic minimum class group of each sample label according to the target classification and the minimum class that is not in the target classification, wherein the minimum class is the classification corresponding to each sample category label in the harmful speech sample set; calculating the classification cut according to the parasitic minimum class group corresponding to each sample; adding the classification cuts of all samples to obtain the total classification cut; calculating the classification accuracy according to the total classification cut and the total maximum classification cut.

[0174] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: calculating the classification cut according to the parasitic minimum class group corresponding to each sample, including: when the predicted category label of the harmful speech sample is the same as the sample category label, calculating the classification cut according to the number of samples in each minimum class in the set of minimum classes and the total number of samples; when the predicted category label of the harmful speech sample is different from the sample category label, calculating the classification cut according to the number of samples in the target classification, the number of samples in each minimum class that is not in the target classification, and the total number of samples.

[0175] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: the classification accuracy includes the total classification cut rate; the classification accuracy is calculated based on the total classification cut amount and the total maximum classification cut amount, including: calculating the maximum classification cut amount based on the minimum class of the sample category label, and calculating the total maximum classification cut amount based on the maximum classification cut amount; calculating the total classification cut rate based on the total classification cut amount and the total maximum classification cut amount.

[0176] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented: the classification accuracy includes the total classification loss rate; the classification accuracy is calculated based on the total classification cut amount and the total maximum classification cut amount, including: calculating the total classification loss amount based on the total classification cut amount and the total maximum classification cut amount; calculating the total classification loss rate based on the total classification loss amount and the total maximum classification cut amount.

[0177] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: a process for determining a classification level, including: taking the classification corresponding to each sample category label in the harmful speech sample set as the minimum class of the classification level; performing classification processing according to the minimum class and a preset iterative classification algorithm to obtain at least one classification set, wherein a classification set contains the classification obtained by one classification; and establishing a classification level according to the association relationship between the minimum class and the classification contained in at least one classification set.

[0178] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0179] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0180] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A classification accuracy evaluation method, characterized in that: The method comprises: Acquire a harmful speech sample set, wherein the harmful speech sample set includes a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample; Inputting each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtaining a predicted category label; In a preset classification hierarchy, searching upward layer by layer and comparing the upper classification of the predicted category label and the upper classification of the sample category label of the harmful speech sample; When the upper-level classification of the predicted category label is the same as the upper-level classification of the sample category label of the harmful speech sample, the same classification is used as the target classification; wherein the target classification is the lowest-level classification including the predicted category label and the sample category label; Determine a parasitic minimum class group for each sample label according to the target classification and the minimum class not in the target classification, wherein the minimum class is the classification corresponding to each sample category label in the harmful speech sample set; Calculate the classification cutoff according to the parasitic minimum group corresponding to each sample; Add up the classification cuts of all samples to get the total classification cut; The classification accuracy is calculated based on the total classification amount and the total maximum classification amount.

2. The method according to claim 1, characterized in that: The calculating of the classification cutoff according to the parasitic minimum cluster group corresponding to each sample includes: When the predicted category label of the harmful speech sample is the same as the sample category label, calculating the classification cutoff according to the number of samples in each smallest class in the set of smallest classes and the total number of samples; When the predicted category label of the harmful speech sample is different from the sample category label, the classification cutoff is calculated according to the number of samples in the target category, the number of samples of each smallest category not in the target category, and the total number of samples.

3. The method according to claim 1, characterized in that The classification accuracy includes the total classification cut rate; the calculation of the classification accuracy based on the total classification cut amount and the total maximum classification cut amount includes: Calculate the maximum classification cut according to the minimum class of the sample category label, and calculate the total maximum classification cut according to the maximum classification cut; The total classification cut rate is calculated according to the total classification cut amount and the total maximum classification cut amount.

4. The method according to claim 1, characterized in that The classification accuracy includes the total classification loss rate; The calculating the classification accuracy according to the total classification amount and the total maximum classification amount includes: Calculate the total classified loss amount according to the total classified cut amount and the total maximum classified cut amount; The total classification loss rate is calculated based on the total classification loss amount and the total maximum classification cut amount.

5. The method according to claim 1, characterized in that The process of determining the classification level includes: Taking the classification corresponding to each sample category label in the harmful speech sample set as the smallest category of the classification level; Performing classification processing according to the minimum class and a preset iterative classification algorithm to obtain at least one classification set, wherein one classification set includes a classification obtained by one classification; The classification hierarchy is established according to the association relationship between the smallest class and the categories included in the at least one classification set.

6. A classification accuracy evaluation device, characterized in that: The device comprises: An acquisition module, used to acquire a harmful speech sample set, wherein the harmful speech sample set includes a plurality of harmful speech samples and a sample category label corresponding to each harmful speech sample; A classification module, used for inputting each harmful speech sample in the harmful speech sample set into the harmful speech classification model to be evaluated for classification, and obtaining a predicted category label; A target classification determination module is used to search and compare the upper classification of the predicted category label and the upper classification of the sample category label of the harmful speech sample in a preset classification hierarchy layer by layer; when the upper classification of the predicted category label is the same as the upper classification of the sample category label of the harmful speech sample, the same classification is used as the target classification; wherein the target classification is the lowest level classification including the predicted category label and the sample category label; A determination module is used to determine the parasitic minimum class group of each sample label according to the target classification and the minimum class not in the target classification, wherein the minimum class is the classification corresponding to each sample category label in the harmful speech sample set; calculate the classification cut according to the parasitic minimum class group corresponding to each sample; add the classification cuts of all samples to obtain a total classification cut; calculate the classification accuracy according to the total classification cut and the total maximum classification cut.

7. The device according to claim 6, characterized in that The determination module is also used to calculate the classification cut according to the number of samples in each smallest class in the set of smallest classes and the total number of samples when the predicted category label of the harmful speech sample is the same as the sample category label; and when the predicted category label of the harmful speech sample is different from the sample category label, calculate the classification cut according to the number of samples in the target classification, the number of samples in each smallest class that is not in the target classification, and the total number of samples.

8. The device according to claim 6, characterized in that The classification accuracy includes the total classification cut rate; the determination module is also used to calculate the maximum classification cut amount based on the minimum class of the sample category label, calculate the total maximum classification cut amount based on the maximum classification cut amount; and calculate the total classification cut rate based on the total classification cut amount and the total maximum classification cut amount.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images