Text classification method, device and computer equipment based on confusion matrix

Abstract

The present application relates to a text classification method, apparatus and computer equipment based on a confusion matrix. First, a confusion matrix of a candidate classification algorithm before being attacked is obtained. Based on the confusion matrix, the centroid between correct and incorrect predictions based on the same true label is introduced. Then, a centroid offset quadrilateral is formed, and the total area of ​​the centroid offset quadrilateral in the confusion matrix before being attacked is calculated. Then, an attack algorithm is used to attack each candidate classification algorithm, and a confusion matrix of the candidate classification algorithm after being attacked is obtained. Similarly, a centroid offset quadrilateral is constructed. By using this method, the difference in the total area of ​​the centroid offset quadrilateral before and after being attacked can be calculated to intuitively see the difference in the robustness of the candidate classification algorithm. The classification algorithm with the best robustness can be scientifically and quickly screened for text classification, thereby ensuring the stability of the text classification effect.

Description

Technical Field

[0001] The present application relates to the technical field of text classification, and in particular to a text classification method, apparatus, and computer device based on a confusion matrix. Background Art

[0002] Equipment intelligence refers to the organic integration of artificial intelligence algorithms into existing equipment systems to enhance their capabilities in recognition, reasoning, judgment, decision-making, control, and environmental adaptation. Traditional equipment testing and evaluation methods struggle to measure the intelligence of intelligent equipment, nor accurately analyze and evaluate its performance and reliability. To address the evaluation of intelligent equipment, it is necessary to assess the machine learning algorithms within it to characterize the equipment's intelligence level. Currently, research on evaluating algorithm intelligence focuses on dimensions such as robustness to adversarial attacks, data security, fairness, and explainability.

[0003] Currently, adversarial robustness evaluation methods are primarily categorized into benchmark evaluation and metric evaluation. The former primarily utilizes adversarial training using different attack and defense algorithms to obtain benchmark rankings and use them to compare adversarial robustness. The latter focuses on the generation process of adversarial examples and proposes a series of adversarial robustness evaluation metrics, making the measurement of adversarial robustness more comprehensive and reasonable. Metrics can be further categorized into model-oriented and data-oriented metrics for each stage, including model input, training, and decision-making. For example, metrics such as adversarial example classification accuracy and average confidence of adversarial categories measure adversarial robustness based on the model's output under adversarial conditions. Metrics such as neuron sensitivity and CLEVER score measure the model's internal response to adversarial examples, thereby measuring the model's adversarial robustness. Metrics such as k-section neuron coverage and average structural similarity indirectly measure the quality of training data from the perspectives of test adequacy and visual imperceptibility, thereby indirectly measuring the model's adversarial robustness. However, these evaluation methods focus on either quantifying or visualizing adversarial robustness, and neither approach simultaneously achieves quantification and visualization.

[0004] The confusion matrix is ​​a summary matrix of the classification results of the intelligent text classification algorithm. Its rows and columns represent the true category and predicted category of the text samples, respectively. The value in each column represents the number of true samples predicted to be of that category. It is widely used in the accuracy evaluation of multi-classification tasks and the performance evaluation of intelligent candidate text classification algorithms. It is one of the important means to measure the adversarial robustness of the output results of intelligent candidate text classification algorithms.

[0005] However, the confusion matrix focuses on quantifying and visualizing the adversarial robustness of a single intelligent candidate text classification algorithm. It has certain limitations in visualizing the differences in adversarial robustness of different intelligent candidate text classification algorithms. It cannot provide an intuitive and quantitative description of the differences. This makes it impossible to quickly and easily compare the performance of multiple intelligent candidate text classification algorithms and select the intelligent device embedded with the candidate text classification algorithm with the best adversarial robustness, which may affect the text classification effect. Take a three-category dataset based on two different classification algorithms a and b, which are subjected to the same attack, as an example. Figure 1 As shown, Figure 1 (a) is the confusion matrix before the attack, Figure 1 (b) represents the confusion matrix of the three-category dataset after being attacked based on text classification algorithm a. Figure 1 (c) represent the confusion matrices of the three classification datasets based on text classification algorithm b after being subjected to the same attack. Figure 1 (a) with Figure 1 (b), and Figure 1 (a) with Figure 1 (c) shows that the confusion matrix can more intuitively and quantitatively describe the adversarial robustness of a single text classification algorithm. Figure 1 (b) with Figure 1 (c) It is not difficult to find that the confusion matrix can only qualitatively explain the differences in the adversarial robustness of different text classifiers, but it cannot intuitively and quantitatively describe the differences. Therefore, it is impossible to intuitively and quantitatively compare the performance differences in adversarial robustness between candidate text classification algorithms, and thus it is impossible to select the best text classification algorithm, and it is difficult to ensure the stability of the text classification effect.

[0006] The text samples can be emails, whose classification labels can include important, unimportant, and uncertain, or recruitment emails, training institution emails, business information emails, advertising emails, subscription emails, and so on. If a text classification algorithm with poor adversarial robustness is attacked, it may misjudge the email category, preventing users from timely accessing the emails they need. For example, while job hunting, users may be more interested in recruitment emails, but existing methods may classify recruitment emails as training institution emails, preventing users from timely accessing information about recruitment emails, causing great inconvenience. Therefore, it is necessary to ensure that text systems such as emails are embedded with text classification algorithms with strong adversarial robustness to meet user needs and optimize the user experience. Summary of the Invention

[0007] Based on this, it is necessary to provide a confusion matrix-based text classification method, device and computer equipment that can quickly screen out the classification algorithm with the best robustness to text classification, thereby improving the stability of text classification effects, in order to address the above technical problems.

[0008] A text classification method based on a confusion matrix, the method comprising:

[0009] Obtain an evaluation text sample set; all evaluation text samples in the evaluation text sample set have corresponding true classification labels;

[0010] A plurality of text classification algorithms are respectively used to predict and classify the evaluation text sample set, and a first confusion matrix corresponding to each text classification algorithm is obtained based on the prediction and classification results, and a first total area of ​​a plurality of centroid offset quadrilaterals in the first confusion matrix is ​​calculated, and a preset number of candidate text classification algorithms are screened out from the plurality of text classification algorithms according to the size of the first total area; in the confusion matrix, the horizontal coordinate of the center point of each square is the true classification label of the evaluation text sample, and the vertical coordinate is the predicted classification label of the evaluation text sample; the weight of the center point of each square is the number of evaluation text samples in the corresponding square; the number of evaluation text samples is obtained by setting the confidence of the original predicted classification label to 1; the number of centroid offset quadrilaterals is , N represents the number of classification labels;

[0011] Among them, the step of calculating the area of ​​the centroid offset quadrilateral includes: determining the construction area of ​​the current centroid offset quadrilateral according to the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values ​​in the confusion matrix, within the construction area, determining the horizontal coordinates of the two offset centroids according to the two true classification labels, determining the vertical coordinates of the two offset centroids according to the vertical coordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels, determining the positions of the two offset centroids according to the horizontal coordinates and vertical coordinates, obtaining the area of ​​the centroid offset quadrilateral according to the two offset centroids and the center points of the squares correctly corresponding to the two predicted classifications, and calculating the area of ​​the centroid offset quadrilateral;

[0012] Using a pre-selected attack algorithm, a deceptive text sample is added to the process of predicting and classifying the evaluation text sample set by each candidate text classification algorithm to obtain a second confusion matrix corresponding to each candidate text classification algorithm, and a second total area of ​​multiple centroid-shifted quadrilaterals in the second confusion matrix is ​​calculated;

[0013] Comparing the adversarial robustness of each candidate text classification algorithm based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral;

[0014] The candidate text classification algorithm with the best adversarial robustness is embedded into intelligent devices to perform target text classification.

[0015] A text classification device based on a confusion matrix, comprising:

[0016] An evaluation text sample set acquisition module is used to acquire an evaluation text sample set; each evaluation text sample in the evaluation text sample set has a corresponding true classification label;

[0017] The first total area calculation module is used to respectively use multiple text classification algorithms to predict and classify the evaluation text sample set, obtain the first confusion matrix corresponding to each text classification algorithm based on the prediction and classification results, calculate the first total area of ​​multiple centroid offset quadrilaterals in the first confusion matrix, and screen out a preset number of candidate text classification algorithms from the multiple text classification algorithms according to the size of the first total area; in the confusion matrix, the horizontal coordinate of the center point of each square is the true classification label of the evaluation text sample, and the vertical coordinate is the predicted classification label of the evaluation text sample; the weight of the center point of each square is the number of evaluation text samples in the corresponding square; the number of evaluation text samples is obtained by setting the confidence of the original predicted classification label to 1; the number of centroid offset quadrilaterals is , N represents the number of classification labels;

[0018] Among them, the step of calculating the area of ​​the centroid offset quadrilateral includes: determining the construction area of ​​the current centroid offset quadrilateral according to the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values ​​in the confusion matrix, within the construction area, determining the horizontal coordinates of the two offset centroids according to the two true classification labels, determining the vertical coordinates of the two offset centroids according to the vertical coordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels, determining the positions of the two offset centroids according to the horizontal coordinates and vertical coordinates, obtaining the area of ​​the centroid offset quadrilateral according to the two offset centroids and the center points of the squares correctly corresponding to the two predicted classifications, and calculating the area of ​​the centroid offset quadrilateral;

[0019] A second total area calculation module is configured to add a deceptive text sample to a process of predicting and classifying the evaluation text sample set using each candidate text classification algorithm using a preselected attack algorithm, obtain a second confusion matrix corresponding to each candidate text classification algorithm, and calculate a second total area of ​​a plurality of centroid-shifted quadrilaterals in the second confusion matrix;

[0020] an adversarial robustness comparison module, configured to compare adversarial robustness of each candidate text classification algorithm based on a difference between a first total area and a second total area of ​​a centroid offset quadrilateral;

[0021] The target text classification module is used to embed the candidate text classification algorithm with the best adversarial robustness into the intelligent device for target text classification.

[0022] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are performed:

[0023] Obtain an evaluation text sample set; all evaluation text samples in the evaluation text sample set have corresponding true classification labels;

[0024] A plurality of text classification algorithms are respectively used to predict and classify the evaluation text sample set, and a first confusion matrix corresponding to each text classification algorithm is obtained based on the prediction and classification results, and a first total area of ​​a plurality of centroid offset quadrilaterals in the first confusion matrix is ​​calculated, and a preset number of candidate text classification algorithms are screened out from the plurality of text classification algorithms according to the size of the first total area; in the confusion matrix, the horizontal coordinate of the center point of each square is the true classification label of the evaluation text sample, and the vertical coordinate is the predicted classification label of the evaluation text sample; the weight of the center point of each square is the number of evaluation text samples in the corresponding square; the number of evaluation text samples is obtained by setting the confidence of the original predicted classification label to 1; the number of centroid offset quadrilaterals is , N represents the number of classification labels;

[0025] Among them, the step of calculating the area of ​​the centroid offset quadrilateral includes: determining the construction area of ​​the current centroid offset quadrilateral according to the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values ​​in the confusion matrix, within the construction area, determining the horizontal coordinates of the two offset centroids according to the two true classification labels, determining the vertical coordinates of the two offset centroids according to the vertical coordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels, determining the positions of the two offset centroids according to the horizontal coordinates and vertical coordinates, obtaining the area of ​​the centroid offset quadrilateral according to the two offset centroids and the center points of the squares correctly corresponding to the two predicted classifications, and calculating the area of ​​the centroid offset quadrilateral;

[0026] Using a pre-selected attack algorithm, a deceptive text sample is added to the process of predicting and classifying the evaluation text sample set by each candidate text classification algorithm to obtain a second confusion matrix corresponding to each candidate text classification algorithm, and a second total area of ​​multiple centroid-shifted quadrilaterals in the second confusion matrix is ​​calculated;

[0027] Comparing the adversarial robustness of each candidate text classification algorithm based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral;

[0028] The candidate text classification algorithm with the best adversarial robustness is embedded into intelligent devices to perform target text classification.

[0029] The above-mentioned text classification method, device, computer equipment and storage medium based on the confusion matrix first obtain the first confusion matrix corresponding to the text classification algorithm, and on the basis of the first confusion matrix, introduce the centroid between the correct prediction and the wrong prediction based on the same true label, and form a centroid offset quadrilateral based on the centroid corresponding to the two classification labels, and calculate the total area of ​​multiple centroid offset quadrilaterals in the first confusion matrix. It can be understood that the total area of ​​the centroid offset quadrilateral corresponding to the first confusion matrix can be used to judge the classification accuracy of the corresponding text classification algorithm. If the classification accuracy is higher, the smaller the sample size of the evaluation text in the square with wrong prediction, the closer the offset centroid is to the center point of the square with correct prediction, and the smaller the area of ​​the corresponding centroid offset quadrilateral. This scheme can first screen out the candidate text classification algorithm set from the initial text classification algorithm set to preliminarily ensure the accuracy of text classification; then, use the attack algorithm to The method attacks each candidate classification algorithm, that is, the attack algorithm generates deceptive text samples, which are input into the candidate text classification algorithm together with the evaluation text sample set, and the second confusion matrix corresponding to the candidate classification algorithm is obtained. After the classification algorithm is attacked, the correctly predicted text samples will be transferred to the wrongly predicted squares. Similarly, by constructing the centroid shift quadrilateral, it can be seen that the worse the adversarial robustness of the classification algorithm, the more text samples are transferred, and the greater the difference in the total area of ​​the centroid shift quadrilateral after the attack and the centroid shift quadrilateral before the attack. Therefore, this method can intuitively see the difference in the adversarial robustness of the candidate classification algorithm by calculating the difference in the total area of ​​the centroid shift quadrilateral before and after the attack, providing a powerful visualization method support for adversarial robustness evaluation, and then scientifically and quickly screening out the classification algorithm with the best adversarial robustness to perform text classification tasks, thereby ensuring the stability of text classification effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The following is a diagram of the confusion matrix of a three-category dataset based on different classification algorithms before and after being attacked by the same attack; Figure 1 (a) is the confusion matrix before the attack, Figure 1 (b) represents the confusion matrix of the three-category dataset after being attacked based on classification algorithm a. Figure 1 (c) represents the confusion matrix of the three-category dataset based on classification algorithm b after being subjected to the same attack;

[0031] Figure 2 This is a flowchart of the text classification method based on confusion matrix;

[0032] Figure 3 Schematic diagram of the confusion matrix for binary classification;

[0033] Figure 4 is the centroid offset quadrilateral based on the N-class confusion matrix, where Figure 4 (a) The center of mass offset quadrilateral before the attack, Figure 4 (b) is the mass center displacement quadrilateral after being attacked;

[0034] Figure 5 Flowchart for visualizing adversarial robustness evaluation based on centroid-shifted quadrilaterals;

[0035] Figure 6 A diagram showing the confusion matrix before and after changing the order of labels for an n-classification problem.

[0036] Figure 7 This is a schematic diagram before and after changing the order of categories in the confusion matrix; Figure 7 (a) is the corresponding centroid offset quadrilateral when the order is 0, 1, 2. Figure 7 (b) is the corresponding centroid offset quadrilateral when the order is 0, 2, 1, Figure 7 (c) is the corresponding centroid offset quadrilateral when the order is 1, 0, 2, Figure 7 (d) is the corresponding centroid offset quadrilateral when the order is 1, 2, 0, Figure 7 (e) is the corresponding centroid offset quadrilateral when the order is 2, 0, 1, Figure 7 ((f) is the corresponding centroid offset quadrilateral when the order is 2, 1, 0;

[0037] Figure 8 Schematic diagram of the correlation analysis between ΔS and Δacc; Figure 8 (a) is the correlation between ΔS and Δacc in the case of a binary classification data set, Figure 8 (b) Correlation between ΔS and Δacc in the case of multi-classification dataset;

[0038] Figure 9 The diagram is a schematic diagram showing the relationship between the area of ​​the centroid offset quadrilateral corresponding to the FGSM and Deepfool attack algorithms based on the intelligent classification algorithm ANN, where: Figure 9 (a) is the centroid offset quadrilateral corresponding to the true classification labels 0 and 1, Figure 9 (b) is the centroid offset quadrilateral corresponding to the true classification labels 0 and 2. Figure 9 (c) is the centroid offset quadrilateral corresponding to the true classification labels 1 and 2;

[0039] Figure 10 The diagram is a diagram showing the relationship between the area of ​​the centroid offset quadrilateral corresponding to the FGSM and Deepfool attack algorithms based on the intelligent classification algorithm LR; Figure 10(a) is the centroid offset quadrilateral corresponding to the true classification labels 0 and 1, Figure 10 (b) is the centroid offset quadrilateral corresponding to the true classification labels 0 and 2. Figure 10 (c) is the centroid offset quadrilateral corresponding to the true classification labels 1 and 2;

[0040] Figure 11 Schematic diagram of the relationship between the size of the centroid offset quadrilateral area corresponding to the FGSM and Deepfool attack algorithms based on the intelligent classification algorithm SVM; Figure 11 (a) is the centroid offset quadrilateral corresponding to the true classification labels 0 and 1, Figure 11 (b) is the centroid offset quadrilateral corresponding to the true classification labels 0 and 2. Figure 11 (c) is the centroid offset quadrilateral corresponding to the true classification labels 1 and 2;

[0041] Figure 12 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] In one embodiment, Figure 2 As shown in FIG, a text classification method based on a confusion matrix is ​​provided, which includes the following steps:

[0044] Step 202: Obtain an evaluation text sample set.

[0045] Among them, the evaluation text samples in the evaluation text sample set all have corresponding true classification labels.

[0046] In step 204, a plurality of text classification algorithms are respectively used to perform predictive classification on the evaluation text sample set, a first confusion matrix corresponding to each text classification algorithm is obtained based on the results of the predictive classification, and a first total area of ​​a plurality of centroid offset quadrilaterals in the first confusion matrix is ​​calculated, and a preset number of candidate text classification algorithms are screened out from the plurality of text classification algorithms based on the size of the first total area.

[0047] In the confusion matrix, the horizontal coordinate of the center point of each square is the true classification label of the evaluation text sample, and the vertical coordinate is the predicted classification label of the evaluation text sample.

[0048] The confusion matrix is ​​a matrix used to summarize the results of an intelligent classification algorithm. It has strong universality and is easy to understand. For k-ary classification, the confusion matrix is ​​a k×k table used to record the prediction results of the intelligent classification algorithm. Taking binary classification as an example, its confusion matrix is ​​a 2×2 table, as shown in the diagram below. Figure 3 As shown. In the confusion matrix, each row represents the true classification label, and each column represents the predicted classification label. The four indicators corresponding to the binary confusion matrix are: TP (True Positive) represents the sample whose true classification label is 0 and the predicted classification label is also 0; FN (False Negative) represents the sample whose true classification label is 0 and the predicted classification label is 1; FP (False Positive) represents the sample whose true classification label is 1 and the predicted classification label is 0; TN (True Negative) represents the sample whose true classification label is 1 and the predicted classification label is also 1. From the confusion matrix, we can get the calculation method of the classification accuracy, precision, recall and other indicators as follows:

[0049]

[0050]

[0051]

[0052] The weight of the center point of each square is the number of evaluation text samples in the corresponding square. The evaluation text sample size is obtained by setting the confidence of the original predicted classification label to 1. In practice, the output of some intelligent classification algorithms is the prediction confidence of each category, and the sample label is then determined to be the category with the highest confidence. For example, when the ANN intelligent classification algorithm classifies a three-category data set with labels 0, 1, and 2, the output corresponding to a sample is the prediction confidence of the three categories [0.3, 0.6, 0.1]. T , so the predicted label of this sample is the label with the highest prediction confidence, 1. In this solution, the confidence of the final predicted label of each text sample is 1 by default, without considering the confidence of classification into other categories.

[0053] It can be understood that in a confusion matrix, the number of centroid offset quadrilaterals is In the confusion matrix of the n-classification problem, since there is only one centroid offset quadrilateral between any two different categories, that is, classification labels, there are a total of A centroid offset quadrilateral.

[0054] Among them, the step of calculating the area of ​​the centroid offset quadrilateral includes: determining the construction area of ​​the current centroid offset quadrilateral according to the squares corresponding to any two true classification labels and two predicted classification labels with the same coordinate values ​​in the confusion matrix, within the construction area, determining the horizontal coordinates of the two offset centroids according to the two true classification labels, determining the vertical coordinates of the two offset centroids according to the vertical coordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels, determining the positions of the two offset centroids according to the horizontal and vertical coordinates of the offset centroids, obtaining the area of ​​the centroid offset quadrilateral according to the two offset centroids and the center points of the squares correctly corresponding to the two predicted classifications, and calculating the area of ​​the centroid offset quadrilateral.

[0055] It can be understood that the total area of ​​the centroid offset quadrilateral corresponding to the first confusion matrix can be used to judge the classification accuracy of the corresponding text classification algorithm. If the classification accuracy is higher, the smaller the sample size of the evaluation text in the wrongly predicted square, the smaller its weight will be. Then the closer the vertical coordinate of the offset centroid is to the vertical coordinate of the center point of the correctly predicted square, the smaller the area of ​​the corresponding centroid offset quadrilateral will be. This scheme can first screen out the candidate text classification algorithm set from the initial text classification algorithm set to preliminarily ensure the accuracy of text classification.

[0056] Step 206 , using a pre-selected attack algorithm, adds the deceptive text sample to the process of predicting and classifying the evaluation text sample set by each candidate text classification algorithm, obtains a second confusion matrix corresponding to each candidate text classification algorithm, and calculates the second total area of ​​multiple centroid offset quadrilaterals in the second confusion matrix.

[0057] The attack algorithm generates deceptive text samples and inputs them into the candidate text classification algorithm together with the evaluation text sample set. At this time, the candidate text classification algorithm may be affected. The worse the adversarial robustness of the candidate text classification algorithm, the more likely the number of evaluation text samples corresponding to the squares in the confusion matrix may be transferred. This is reflected in the fact that the fewer the evaluation text samples in the correctly predicted squares, the lower their weights, the farther the vertical coordinate of the offset centroid is from the vertical coordinate of the center point of the predicted correct square, and the larger the area of ​​the corresponding centroid offset quadrilateral. Therefore, the difference in the total area of ​​the centroid offset quadrilateral can reflect the strength of the adversarial robustness of the candidate text classification algorithm, that is, the strength of its ability to maintain the stability of its text classification accuracy.

[0058] Step 208 : Compare the adversarial robustness of each candidate text classification algorithm based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral.

[0059] In this scheme, the number of pre-selected attack algorithms can be one or more types. When there is only one attack algorithm, the candidate classification algorithm corresponding to the largest total area difference is directly selected to classify the target text; when there are multiple attack algorithms, each candidate text classification algorithm and each attack algorithm has a corresponding total area difference. If it is necessary to comprehensively consider the robustness of the candidate classification algorithm against multiple attack algorithms, it is possible to consider calculating the weighted sum of the total area differences of each candidate text classification algorithm, and the weight of the more important attack algorithm can be set higher.

[0060] As can be understood, using the difference in the total area of ​​the quadrilaterals with centroid offsets before and after an attack as a visual robustness evaluation metric can more intuitively reflect the differences in robustness between classification algorithms. This allows for rapid selection of the most robust classification algorithm for embedding in intelligent devices for text classification tasks, ensuring the stability of text classification results. Furthermore, using the difference in the total area of ​​the quadrilaterals with centroid offsets before and after an attack as a visual robustness evaluation metric can also intuitively reflect the differences in robustness of classification algorithms against different attack algorithms.

[0061] Step 210: embed the candidate classification algorithm with the best adversarial robustness into the intelligent device to perform target text classification.

[0062] The centroid shift quadrilateral proposed in this scheme is mainly generated based on four squares consisting of two different true classification labels and the corresponding two predicted classification labels in the confusion matrix. The centroid shift quadrilateral based on the N-classification confusion matrix is ​​as follows Figure 4 As shown, Figure 4 (a) The center of mass offset quadrilateral before the attack, Figure 4 (b) is the center of mass offset quadrilateral after being attacked.

[0063] In the above-mentioned text classification method based on confusion matrix, first, the first confusion matrix corresponding to the text classification algorithm is obtained, and on the basis of the first confusion matrix, the centroid between the correct prediction and the wrong prediction based on the same true label is introduced, and the centroid corresponding to the two classification labels is used as the basis to form a centroid offset quadrilateral, and the total area of ​​multiple centroid offset quadrilaterals in the first confusion matrix is ​​calculated. It can be understood that the total area of ​​the centroid offset quadrilateral corresponding to the first confusion matrix can be used to judge the classification accuracy of the corresponding text classification algorithm. If the classification accuracy is higher, the smaller the sample size of the evaluation text in the square with wrong prediction, the closer the offset centroid is to the center point of the square with correct prediction, and the smaller the area of ​​the corresponding centroid offset quadrilateral. This scheme can first screen out the candidate text classification algorithm set from the initial text classification algorithm set to preliminarily ensure the accuracy of text classification; then, the attack algorithm is used to attack each candidate A classification algorithm is selected for attack, that is, the attack algorithm generates deceptive text samples, which are input into the candidate text classification algorithm together with the evaluation text sample set, and the second confusion matrix corresponding to the candidate classification algorithm is obtained. After the classification algorithm is attacked, the correctly predicted text samples will be transferred to the wrongly predicted squares. Similarly, by constructing the centroid shift quadrilateral, it can be seen that the worse the adversarial robustness of the classification algorithm, the more text samples are transferred, and the greater the difference in the total area of ​​the centroid shift quadrilateral after the attack and the centroid shift quadrilateral before the attack. Therefore, this method can be used to intuitively see the difference in the adversarial robustness of the candidate classification algorithm by calculating the difference in the total area of ​​the centroid shift quadrilateral before and after the attack, providing a powerful visualization method support for adversarial robustness evaluation, and then scientifically and quickly screening out the classification algorithm with the best adversarial robustness to perform text classification tasks, thereby ensuring the stability of text classification effects.

[0064] In one embodiment, in the first confusion matrix, the step of constructing a centroid offset quadrilateral includes:

[0065] Determine the construction area of ​​the current centroid offset quadrilateral according to the line connecting the center points of the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values;

[0066] In the build area:

[0067] The horizontal coordinates of the center points of the two squares are used as the horizontal coordinates of the two offset centroids:

[0068] X i =x i ;

[0069] X j =x j ;

[0070] Among them, X iRepresents the offset centroid M of the true classification label i i The horizontal coordinate, x i Indicates the horizontal coordinate of the center point of the square with the true classification label i, X j Indicates the offset centroid M of the true classification label j j The horizontal coordinate, x j The horizontal coordinate of the center point of the square whose true classification label is j;

[0071] The sample size of the evaluation text in the grid is used as the weight of the center point of the corresponding grid;

[0072] The ordinates of the two offset centroids are determined based on the ordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels:

[0073]

[0074]

[0075] Among them, Y i Represents the offset centroid M of the true classification label i i The vertical coordinate, Y j Indicates the offset centroid M of the true classification label j j The vertical coordinate, y j The vertical coordinate of the center point of the square whose true classification label is j, y i The vertical coordinate of the center point of the square whose true classification label is i, C ij C represents the number of evaluation text samples in the square with the true classification label i and the predicted classification label j, ji C represents the number of evaluation text samples in the square with the true classification label j and the predicted classification label i. jj Indicates the number of evaluation text samples in the square where the true classification label is j and the predicted classification label is also j, C ii Indicates the number of evaluation text samples in the square where the true classification label is i and the predicted classification label is also i.

[0076] The centroid offset quadrilateral is constructed according to the two horizontal coordinates and two vertical coordinates of the offset centroid.

[0077] In one embodiment, calculating a first total area of ​​a plurality of centroid-shifted quadrilaterals in a first confusion matrix includes:

[0078] The first total area of ​​the multiple centroid offset quadrilaterals in the first confusion matrix is ​​calculated as:

[0079]

[0080] Among them, S i,jThe first area of ​​the centroid offset quadrilateral in the four squares where the true classification labels are i, j and the predicted classification labels are also i, j, Denominator represents the area of ​​the quadrilateral formed by the center points of the four squares corresponding to the centroid offset quadrilateral. The role of is to eliminate the interference of the position relationship between the two categories on the centroid offset quadrilateral.

[0081] In one embodiment, in the second confusion matrix, the step of constructing a centroid offset quadrilateral includes:

[0082] Determine the construction area of ​​the current centroid offset quadrilateral according to the line connecting the center points of the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values;

[0083] In the build area:

[0084] The horizontal coordinates of the center points of the two squares are used as the horizontal coordinates of the two offset centroids:

[0085] X i '=x j ;

[0086] X j '=x j ;

[0087] Among them, X i 'Indicates the offset centroid M of the true classification label i i 'The horizontal coordinate, X j 'Indicates the offset centroid M of the true classification label j j ' horizontal axis;

[0088] The number of evaluation text samples in a square is used as the weight of the center point of the corresponding square; in the second confusion matrix, the number of evaluation text samples in the square is calculated by the number of evaluation text samples in the corresponding square in the first confusion matrix and the amount of evaluation text sample transfer that changes relative to the predicted classification label in the first confusion matrix;

[0089] The ordinates of the two offset centroids are determined based on the ordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels:

[0090]

[0091]

[0092] Among them, Y i 'Indicates the offset centroid M of the true classification label i i 'Y coordinate, Y j'Indicates the offset centroid M of the true classification label j j 'The vertical coordinate, Δ ij Indicates the amount of evaluation text sample transfer when the predicted classification label of the true classification label is i changes from i to j, Δ ji Indicates the amount of evaluation text sample transfer when the predicted classification label of the true classification label is j changes from j to i;

[0093] A centroid offset quadrilateral is constructed based on the two horizontal coordinates and two vertical coordinates of the offset centroid.

[0094] In one embodiment, calculating a second total area of ​​a plurality of centroid-shifted quadrilaterals in a second confusion matrix includes:

[0095] The second total area of ​​the multiple centroid-shifted quadrilaterals in the second confusion matrix is ​​calculated as:

[0096]

[0097] Among them, S i,j ' represents the second area of ​​the centroid offset quadrilateral in the four squares whose true classification labels are i, j and whose predicted classification labels are also i, j.

[0098] From the above formula, we can see that the greater the impact of the attack algorithm on the intelligent classification algorithm, the greater the sample transfer amount Δ ij and ji The larger the sum, that is, the greater the total offset of the center of mass relative to the center point of the correct square predicted by the classification, the larger the area of ​​the corresponding center of mass offset quadrilateral after the attack, and the greater the difference from the area of ​​the center of mass offset quadrilateral before the attack.

[0099] In one embodiment, comparing the adversarial robustness of each candidate text classification algorithm based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral includes:

[0100] The adversarial robustness of each candidate text classification algorithm is compared based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral:

[0101]

[0102] Among them, the smaller the difference, the better the adversarial robustness.

[0103] In one embodiment, the candidate classification algorithm with the best adversarial robustness is embedded into an intelligent device to perform target text classification, including:

[0104] The candidate classification algorithm with the smallest difference between the first total area and the second total area of ​​the centroid offset quadrilateral is embedded into the intelligent device to classify the target text.

[0105] This solution can also be used for sample classification in other fields such as images, audio and video.

[0106] It should be understood that although Figure 2 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 2 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0107] The following experiments are designed by randomly combining different data sets, intelligent classification algorithms, and attack algorithms. Then, based on the centroid offset quadrilateral visualization evaluation metric proposed above, the differences in the adversarial robustness of intelligent classification algorithms against different attack algorithms, as well as the differences in the adversarial robustness between intelligent classification algorithms, are explored and analyzed.

[0108] Step 1: Verify the effectiveness of visualization evaluation indicators through experiments.

[0109] Step 2: Under the same data set and intelligent classification algorithm conditions, different attack algorithms are used to attack the intelligent classification algorithm. Based on the difference in the total area of ​​the centroid offset quadrilateral before and after the attack, the difference in the intelligent classification algorithm's robustness against different attack algorithms is obtained through comparative analysis.

[0110] Step 3: Under the conditions of the same dataset and attack algorithm, the difference in the adversarial robustness of different intelligent classification algorithms is obtained through comparative analysis based on the difference in the total area of ​​the quadrilateral with centroid offset before and after the attack.

[0111] Step 4: Based on the relative positional relationship of the center of mass offset quadrilateral before and after the attack, as well as the differences in shape and area, explore and analyze the movement trends and characteristics of samples of each category after the intelligent classification algorithm is attacked.

[0112] Step 5: Analyze and summarize the above experimental results and draw conclusions.

[0113] The visual evaluation process of adversarial robustness based on centroid-shifted quadrilaterals is as follows: Figure 5 shown.

[0114] 1 Experimental design and result analysis

[0115] 1.1 Experimental Design

[0116] The dataset and attack algorithm used in the experiment are shown below.

[0117] 1.1.1 Experimental Environment

[0118] 1.1.2 Dataset and Attack Algorithm

[0119] The name information of the datasets and algorithms used in the experiment are shown in Table 1.

[0120] Table 1 Dataset and algorithm information

[0121]

[0122] 1.2 Results Analysis

[0123] First, using the wine dataset as an example, we examine the effectiveness of visualization metrics. Then, based on the centroid-shifted quadrilateral adversarial robustness visualization metric, we explore the differences in adversarial robustness across different datasets, intelligent classification algorithms, and attack algorithm combinations. Furthermore, we explore the shift in sample size within each category of the confusion matrix during classification, providing a highly intuitive and quantifiable visualization method for adversarial robustness assessment.

[0124] 1.2.1 Validity test of visualization indicators

[0125] In order to ensure that the proposed visualization indicators can correctly and effectively reflect the changes in adversarial robustness before and after adversarial attacks, which are evaluated by classification accuracy, it is necessary to test the effectiveness of the indicators.

[0126] First, we conduct a feasibility analysis to test whether the order of labels in the confusion matrix will affect the area of ​​the centroid offset quadrilateral. The following diagram shows the confusion matrix before and after changing the order of labels based on the n-classification problem: Figure 6 shown.

[0127] Before exchanging labels, the area S of the quadrilateral with the center of mass offset before and after the attack i,j With S i,j 'See formula (1) and formula (2).

[0128] Swap the labels i and j in the confusion matrix. The centroid offset quadrilateral area S before the attack j,i for:

[0129]

[0130] The center of mass after the attack is offset by the quadrilateral area S ji 'for:

[0131]

[0132] By comparison, we can see that S i,j =Sj,i , S i,j '=S j,i ', then the difference in the area of ​​the quadrilateral with the center of mass offset before and after the attack before exchanging labels is ΔS i,j =S i,j 'S i,j , the difference in the area of ​​the quadrilateral with the center of mass offset before and after the attack after exchanging labels ΔS j,i ×S j,i 'S j,i , so ΔS i,j =ΔS j,i , by all ΔS i,j and ΔS j,i Sum them up separately to get the total area difference ΔS before exchanging labels and ΔS after exchanging labels. change Equal, that is, ΔS change S, the difference in the area of ​​the centroid offset quadrilateral before and after the attack does not change with the change of the order of the confusion matrix labels, and the visualization effect is the same as before changing the order of the labels.

[0133] Take the wine dataset based on ANN intelligent classification algorithm under the attack of FGSM algorithm as an example for experimental verification. The labels of the three-class wine dataset are 0, 1, and 2. Changing the order of categories in the confusion matrix, there are a total of The corresponding centroid offset quadrilateral is as follows: Figure 7 As shown. Among them, Figure 7 (a) is the corresponding centroid offset quadrilateral when the order is 0, 1, 2. Figure 7 (b) is the corresponding centroid offset quadrilateral when the order is 0, 2, 1, Figure 7 (c) is the corresponding centroid offset quadrilateral when the order is 1, 0, 2, Figure 7 (d) is the corresponding centroid offset quadrilateral when the order is 1, 2, 0, Figure 7 (e) is the corresponding centroid offset quadrilateral when the order is 2, 0, 1, Figure 7 (f) is the corresponding centroid offset quadrilateral when the order is 2, 1, and 0. The quantitative calculation values ​​of the centroid offset area under different arrangement orders are shown in Table 3.

[0134] As shown in Table 2, compared with the normal arrangement order 0, 1, and 2, the area of ​​the centroid offset quadrilateral between the same two categories does not change in other arrangement orders. Therefore, the difference in the area of ​​the centroid offset quadrilateral before and after each attack remains unchanged, and thus the difference in the total area of ​​the centroid offset quadrilateral before and after the attack does not change.

[0135] Table 2 Feasibility test of wine dataset based on ANN and FGSM

[0136]

[0137] Therefore, the adversarial robustness visualization indicator based on centroid offset quadrilateral is not affected by the order of label arrangement, and can maintain the evaluation effect when the order of label arrangement changes randomly, which has certain feasibility.

[0138] Then, a correlation test is performed between ΔS and Δacc. The classification accuracy acc based on the confusion matrix is ​​an important indicator for evaluating the robustness of intelligent classification algorithms. It is calculated as the ratio of the number of correctly classified samples to the total number of samples. In the n-classification problem, the calculation method is:

[0139]

[0140] The difference in the total area of ​​the centroid offset quadrilateral before and after the attack is intended to intuitively and quantifiably reflect the changing trend of the classification accuracy before and after the adversarial attack, while also improving the interpretability of the classification results. Therefore, it is necessary to verify the correlation between the classification accuracy and the area of ​​the centroid offset quadrilateral to demonstrate that the proposed visualization indicator is accurate and effective. The difference in classification accuracy before and after the adversarial attack, Δacc, is:

[0141]

[0142] The difference ΔS in the total area of ​​the quadrilateral with respect to the center of mass offset before and after the attack is:

[0143]

[0144] The Δacc and ≥S corresponding to different data sets, attack algorithms, and intelligent classification algorithm combinations are calculated respectively. The specific results are shown in Table 3. Assume that the side length of each square in the confusion matrix is ​​1.

[0145] Table 3 Δacc and ΔS of different dataset and algorithm combinations

[0146]

[0147] Based on Table 3, the relationship between ΔS and Δacc is explored. The correlation analysis between the two is as follows: Figure 8 As shown. Among them, Figure 8 The four fitted lines in (a) overlap. For binary classification problems, since the samples can only be misclassified into another category, the curves corresponding to the binary classification problems are all straight lines with a slope of 45 degrees, such as Figure 8 As shown in (a). Figure 8 In (b), since the number of categories is greater, the number of centroid offset quadrilaterals generated by the confusion matrix is ​​greater, and the corresponding total area difference ΔS is greater, so the slope is greater. Figure 8It can be seen that ΔS and Δacc are positively correlated. The larger ΔS is, the larger Δacc is, indicating that ΔS can quantify the changing trend of adversarial robustness. Therefore, the adversarial robustness visualization indicator based on the centroid offset quadrilateral can accurately and effectively visualize the difference in adversarial robustness before and after the attack, passing the correlation test.

[0148] In summary, the adversarial robustness visualization evaluation index based on centroid offset quadrilateral has passed the effectiveness test, proving the effectiveness of the visualization index proposed in this chapter.

[0149] 1.2.2 Comparative Analysis of Adversarial Robustness Based on Centroid-Shifted Quadrilaterals

[0150] After conducting validation tests, we analyzed the robustness of different intelligent classification algorithms, and of these algorithms against different attack algorithms, using the three-class iris dataset as the research object. We used the centroid-shifted quadrilateral-based robustness visualization metric as a key visualization support method for the conclusions. We also conducted experimental analysis and verification. Graphs for the remaining datasets are provided in the Appendix.

[0151] For the binary classification problem, each confusion matrix has only one centroid offset quadrilateral, which is more intuitive. For the multi-classification problem with n categories (n≥3), each confusion matrix has The centroid-shifted quadrilaterals may have large overlapping areas, making it difficult to directly compare the total area of ​​the centroid-shifted quadrilaterals. Therefore, it is necessary to compare and analyze the centroid-shifted quadrilaterals between the same two categories under different attack algorithms. Assume that the side length of each square in the confusion matrix is ​​1.

[0152] Δacc, ΔS and ΔS of the three-class iris dataset i,j As shown in Table 4.

[0153] Table 4 shows that, first, a horizontal comparison of the ΔS corresponding to the FGSM and Deepfool attack algorithms for the same intelligent classification algorithms is presented. The ΔS corresponding to the ANN is 0.560 and 1.800, respectively; the ΔS corresponding to the LR is 0.520 and 0.910, respectively; and the ΔS corresponding to the SVM is 0.500 and 0.990, respectively. It can be seen that for each intelligent classification algorithm, the ΔS corresponding to the Deepfool algorithm is greater than that of the FGSM algorithm. A larger ΔS indicates a larger Δacc, which in turn indicates greater robustness of the intelligent classification algorithm against the attack algorithm. Therefore, the intelligent classification algorithms ANN, LR, and SVM on the cancer dataset are all more robust against the FGSM attack algorithm than the Deepfool attack algorithm.

[0154] Next, a longitudinal comparison of the ΔS values ​​of the three intelligent classification algorithms under FGSM and Deepfool attacks reveals that under FGSM, the ΔS value for the ANN is 0.560, which is larger than the 0.520 value for LR and the 0.500 value for SVM. Under Deepfool, the ΔS value for the ANN is 1.800, which is larger than the 0.990 value for SVM and the 0.910 value for LR. A larger ΔS value indicates a larger Δacc value, indicating that the ANN has the weakest adversarial robustness, while the SVM and LR have similar adversarial robustness.

[0155] Table 4 Δacc, ΔS and ΔSi,j of the iris dataset

[0156]

[0157] Figures 9 to 11 The relationship between the size of the centroid offset quadrilateral area corresponding to the FGSM and Deepfool attack algorithms based on different intelligent classification algorithms is shown. The overlapping part of the two areas is brown, and the values ​​in brackets represent the centroid offset quadrilateral area corresponding to the FGSM and Deepfool attack algorithms, respectively.

[0158] from Figures 9 to 11 It can be seen that the difference in the area of ​​each centroid offset quadrilateral before and after the Deepfool attack is no less than the difference in the area of ​​each centroid offset quadrilateral before and after the FGSM attack, where Figures 9 to 11 In sub-figure (c), the difference in the area of ​​each centroid offset quadrilateral before and after the Deepfool attack is significantly larger than that of the FGSM, providing an intuitive and effective visualization method to support the conclusion that the intelligent classification algorithm is more robust against the FGSM attack algorithm than the Deepfool algorithm.

[0159] At the same time, by comparison Figure 10 and Figure 11 From the side lengths of the centroid shift quadrilaterals corresponding to FGSM and Deepfool in subfigure (c), we can see that compared with FGSM, the number of samples misclassified from label 1 to label 2, and the number of samples misclassified from label 2 to label 1, both increase significantly after the Deepfool attack. In addition, the corresponding centroid shift quadrilaterals tend to be more square, indicating that the probability of misclassification of samples with label 1 and label 2 is more uniform and random. Figure 10 The center of mass offset quadrilateral corresponding to Deepfool in (b) is a right triangle that is slightly to the upper left of the center of mass offset quadrilateral before the attack. This is to illustrate that the Deepfool attack causes all samples with label 0 to be misclassified as label 2. Figure 11 (c) shows that the Deepfool attack causes all samples with label 2 to be misclassified as label 1, which reflects the randomness of the Deepfool attack algorithm.

[0160] Therefore, the visualization metric based on the centroid shift quadrilateral provides a powerful visualization support for comparing the adversarial robustness of intelligent classification algorithms, as well as for comparative analysis of their robustness against different attack algorithms. It also demonstrates the characteristics of sample shifts under different attack algorithms, providing interpretability for classification results.

[0161] In summary, the visualization evaluation metric based on the centroid shift quadrilateral is highly intuitive and quantifiable for demonstrating differences in adversarial robustness, providing an effective visualization method for comparing the adversarial robustness of ANNs, LRs, and SVMs. Furthermore, the shape of the corresponding centroid shift quadrilateral after the attack and its relative position to before the attack can provide a certain degree of interpretability for the classification results.

[0162] In one embodiment, a text classification apparatus based on a confusion matrix is ​​provided, wherein:

[0163] An evaluation text sample set acquisition module is used to acquire an evaluation text sample set; each evaluation text sample in the evaluation text sample set has a corresponding true classification label;

[0164] The first total area calculation module is used to respectively use multiple text classification algorithms to predict and classify the evaluation text sample set, obtain the first confusion matrix corresponding to each text classification algorithm based on the prediction and classification results, calculate the first total area of ​​multiple centroid offset quadrilaterals in the first confusion matrix, and screen out a preset number of candidate text classification algorithms from the multiple text classification algorithms according to the size of the first total area; in the confusion matrix, the horizontal coordinate of the center point of each square is the true classification label of the evaluation text sample, and the vertical coordinate is the predicted classification label of the evaluation text sample; the weight of the center point of each square is the number of evaluation text samples in the corresponding square; the number of evaluation text samples is obtained by setting the confidence of the original predicted classification label to 1; the number of centroid offset quadrilaterals is , N represents the number of classification labels;

[0165] Among them, the step of calculating the area of ​​the centroid offset quadrilateral includes: determining the construction area of ​​the current centroid offset quadrilateral according to the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values ​​in the confusion matrix, within the construction area, determining the horizontal coordinates of the two offset centroids according to the two true classification labels, determining the vertical coordinates of the two offset centroids according to the vertical coordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels, determining the positions of the two offset centroids according to the horizontal coordinates and vertical coordinates, obtaining the area of ​​the centroid offset quadrilateral according to the two offset centroids and the center points of the squares correctly corresponding to the two predicted classifications, and calculating the area of ​​the centroid offset quadrilateral;

[0166] A second total area calculation module is configured to add a deceptive text sample to a process of predicting and classifying the evaluation text sample set using each candidate text classification algorithm using a preselected attack algorithm, obtain a second confusion matrix corresponding to each candidate text classification algorithm, and calculate a second total area of ​​a plurality of centroid-shifted quadrilaterals in the second confusion matrix;

[0167] an adversarial robustness comparison module, configured to compare adversarial robustness of each candidate text classification algorithm based on a difference between a first total area and a second total area of ​​a centroid offset quadrilateral;

[0168] The target text classification module is used to embed the candidate text classification algorithm with the best adversarial robustness into the intelligent device for target text classification.

[0169] The specific limitations of the confusion matrix-based text classification device can be found in the limitations of the confusion matrix-based text classification method described above and will not be repeated here. Each module in the confusion matrix-based text classification device described above can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the modules can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in a computer device in software form, so that the processor can call and execute the operations corresponding to each of the modules.

[0170] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 12As shown. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store data such as classification algorithms. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a text classification method based on a confusion matrix is ​​implemented.

[0171] Those skilled in the art will understand that Figure 12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0172] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps of the method in the above embodiment when executing the computer program.

[0173] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in the above embodiment are implemented.

[0174] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0175] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0176] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A text classification method based on confusion matrix, characterized in that: The method comprises: Obtain an evaluation text sample set; the evaluation text samples in the evaluation text sample set all have corresponding true classification labels; use a variety of text classification algorithms to predict and classify the evaluation text sample set, obtain a first confusion matrix corresponding to each text classification algorithm based on the prediction and classification results, calculate a first total area of ​​a plurality of centroid offset quadrilaterals in the first confusion matrix, and screen a preset number of candidate text classification algorithms from a variety of text classification algorithms according to the size of the first total area; in the confusion matrix, the horizontal coordinate of the center point of each square is the true classification label of the evaluation text sample, and the vertical coordinate is the predicted classification label of the evaluation text sample; the weight of the center point of each square is the number of evaluation text samples in the corresponding square; the number of evaluation text samples is obtained by setting the confidence of the original predicted classification label to 1; the number of centroid offset quadrilaterals is N represents the number of classification labels; wherein, the step of calculating the area of ​​the centroid offset quadrilateral includes: determining the construction area of ​​the current centroid offset quadrilateral according to the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values ​​in the confusion matrix, within the construction area, determining the horizontal coordinates of the two offset centroids according to the two true classification labels, determining the vertical coordinates of the two offset centroids according to the vertical coordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels, determining the positions of the two offset centroids according to the horizontal and vertical coordinates of the offset centroids, and determining the positions of the two offset centroids according to the horizontal coordinates and vertical coordinates of the two offset centroids and the two predicted classification labels. Measure the center point of the square corresponding to the correct classification, obtain the area of ​​the centroid offset quadrilateral, and calculate the area of ​​the centroid offset quadrilateral; use a pre-selected attack algorithm to add deceptive text samples in the process of predicting and classifying the evaluation text sample set by each candidate text classification algorithm, obtain a second confusion matrix corresponding to each candidate text classification algorithm, and calculate the second total area of ​​multiple centroid offset quadrilaterals in the second confusion matrix; compare the adversarial robustness of each candidate text classification algorithm based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral; embed the candidate text classification algorithm with the best adversarial robustness into the intelligent device to perform target text classification.

2. The method according to claim 1, characterized in that In the first confusion matrix, the step of constructing a centroid offset quadrilateral includes: The current centroid offset quadrilateral construction area is determined based on the line connecting the center points of the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values; within the construction area: the abscissas of the center points of the two squares are used as the abscissas of the two offset centroids respectively: ； ； in, Indicates that the true classification label is The offset centroid The horizontal axis, Indicates that the true classification label is The horizontal coordinate of the center point of the square, Indicates that the true classification label is The offset centroid The horizontal axis, Indicates that the true classification label is The horizontal coordinate of the center point of the square; The sample size of the evaluation text in the grid is used as the weight of the center point of the corresponding grid; The ordinates of the two offset centroids are determined based on the ordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels: ； ； in, Indicates that the true classification label is The offset centroid The vertical coordinate, Indicates that the true classification label is The offset centroid The vertical coordinate, Indicates that the true classification label is The vertical coordinate of the center point of the square, Indicates that the true classification label is The vertical coordinate of the center point of the square, Indicates that the true classification label is , the predicted classification label is The sample size of the evaluation text in the grid, Indicates that the true classification label is , the predicted classification label is The sample size of the evaluation text in the grid, Indicates that the true classification label is , the predicted classification label is also The sample size of the evaluation text in the grid, Indicates that the true classification label is , the predicted classification label is also The sample size of the evaluation text in the grid; The centroid offset quadrilateral is constructed according to the two horizontal coordinates and two vertical coordinates of the offset centroid.

3. The method according to claim 2, characterized in that Calculating a first total area of ​​a plurality of centroid offset quadrilaterals in the first confusion matrix includes: The first total area of ​​the plurality of centroid offset quadrilaterals in the first confusion matrix is ​​calculated as: ； in, The true classification labels are , , and the predicted classification labels are , The first area of ​​the quadrilateral with the centroid of the 4 squares offset, Represents the area of ​​the quadrilateral formed by the center points of the four squares corresponding to the centroid offset quadrilateral.

4. The method according to claim 3, characterized in that In the second confusion matrix, the step of constructing the centroid offset quadrilateral includes: The current centroid offset quadrilateral construction area is determined based on the line connecting the center points of the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values; within the construction area: the abscissas of the center points of the two squares are used as the abscissas of the two offset centroids respectively: ； ； in, Indicates that the true classification label is The offset centroid The horizontal axis, Indicates that the true classification label is The offset centroid The horizontal axis of The number of evaluation text samples in a square is used as the weight of the center point of the corresponding square; in the second confusion matrix, the number of evaluation text samples in the square is calculated by the number of evaluation text samples in the corresponding square in the first confusion matrix and the amount of evaluation text sample transfer that changes relative to the predicted classification label in the first confusion matrix; The ordinates of the two offset centroids are determined based on the ordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels: ； ； in, Indicates that the true classification label is The offset centroid The vertical coordinate, Indicates that the true classification label is The offset centroid The vertical coordinate, Indicates that the true classification label is The predicted classification labels are becomes The amount of evaluation text sample transfer, Indicates that the true classification label is The predicted classification labels are becomes The amount of evaluation text sample transfer; A centroid offset quadrilateral is constructed based on the two horizontal coordinates and two vertical coordinates of the offset centroid.

5. The method according to claim 4, characterized in that Calculating a second total area of ​​a plurality of centroid-shifted quadrilaterals in the second confusion matrix includes: The second total area of ​​the plurality of centroid offset quadrilaterals in the second confusion matrix is ​​calculated as: ； in, The true classification labels are , , and the predicted classification labels are , The centroids of the four squares are offset by the second area of ​​the quadrilateral.

6. The method according to claim 5, characterized in that The adversarial robustness of each candidate text classification algorithm is compared based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral, including: The adversarial robustness of each candidate text classification algorithm is compared based on the difference between the first total area and the second total area of ​​the centroid offset quadrilateral: ； Among them, the smaller the difference, the better the adversarial robustness.

7. The method according to claim 5, characterized in that Embed the candidate classification algorithm with the best adversarial robustness into intelligent devices for target text classification, including: The candidate classification algorithm with the smallest difference between the first total area and the second total area of ​​the centroid offset quadrilateral is embedded into the intelligent device to classify the target text.

8. A text classification device based on confusion matrix, characterized in that The device comprises: An evaluation text sample set acquisition module is used to acquire an evaluation text sample set; the evaluation text samples in the evaluation text sample set all have corresponding true classification labels; a first total area calculation module is used to respectively use a plurality of text classification algorithms to perform prediction classification on the evaluation text sample set, obtain a first confusion matrix corresponding to each text classification algorithm based on the prediction classification result, and calculate the first total area of ​​a plurality of centroid offset quadrilaterals in the first confusion matrix, and screen out a preset number of candidate text classification algorithms from a plurality of text classification algorithms according to the size of the first total area; in the confusion matrix, the horizontal coordinate of the center point of each square is the true classification label of the evaluation text sample, and the vertical coordinate is the predicted classification label of the evaluation text sample; the weight of the center point of each square is the number of evaluation text samples in the corresponding square; the number of evaluation text samples is obtained by setting the confidence of the original predicted classification label to 1; the number of centroid offset quadrilaterals is N represents the number of classification labels; wherein, the step of calculating the area of ​​the centroid offset quadrilateral includes: determining the construction area of ​​the current centroid offset quadrilateral according to the squares corresponding to any two true classification labels and the two predicted classification labels with the same coordinate values ​​in the confusion matrix, within the construction area, determining the horizontal coordinates of the two offset centroids according to the two true classification labels, determining the vertical coordinates of the two offset centroids according to the vertical coordinates and weights of the center points of the two squares corresponding to each true classification label and its corresponding two predicted classification labels, determining the positions of the two offset centroids according to the horizontal coordinates and vertical coordinates, and obtaining the center points of the two offset centroids and the two predicted classifications correctly corresponding squares. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module, a second detection module and a second detection module. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module and a second detection module. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module and a second detection module. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module and a second detection module. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module and a second detection module. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module and a second detection module. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module and a second detection module. The invention relates to an intelligent text classification system comprising: a first detection module, a first detection module and a second detection module.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

Images