A method for classifying easily confused categories based on a two-stage model

Abstract

A method for classifying easily confused categories based on a two-stage model, the technical solution is to construct a two-stage classification system consisting of a preprocessing module, a first-stage classification module, an easily confused category identification module, a boundary sample screening module and a second-stage classification module. A training set and a test set are constructed, and the preprocessing module maps the text or source code in the training set and the test set to a word-level coding sequence set; the first-stage classification module classifies the word-level coding sequence set according to the category table to obtain a confusion matrix; the easily confused category identification module uses the confusion matrix to obtain an easily confused category binary sequence; the two-stage classification system is trained, and the trained two-stage classification system performs preprocessing, first-stage classification, easily confused category identification and boundary sample identification on the target sample. If it is a boundary sample, the second-stage classification module is used for reclassification to obtain a classification result. The classification accuracy of easily confused categories can be improved by using the present invention.

Description

Technical Field

[0001] The present invention relates to the problem of easily confused classification in classification tasks related to the software engineering field, and in particular to a method for classifying easily confused categories based on a two-stage model. Background Art

[0002] Classification tasks are a hot topic in the computer field, covering multiple fields such as computer vision, speech recognition, and natural language processing. In the field of software engineering, classification tasks are also very common, such as vulnerability detection, software requirements classification, code author attribution, and algorithm classification. With the rapid development of technologies such as deep neural networks and data mining, automated classification methods have been widely used in the field of software engineering. For example, classification methods based on deep learning models such as LSTM (see the literature "Long short-term memory", 1997) and BERT (see the literature "Bert: Pre-training of deep bidirectional transformers for language understanding", 2018, Bert: Deep bidirectional transformer pre-training model for language understanding) can better process complex software engineering data and mine deeper features and patterns by using the powerful representation and learning capabilities of deep neural networks, thereby achieving more accurate classification.

[0003] The application of automated classification methods enables software development, testing, and maintenance in the software engineering field to be more efficient. Accurate classification results can help developers better understand and analyze the characteristics and behaviors of software systems, thereby improving software quality and performance. In addition, automated classification methods also play an important auxiliary role in decision-making and problem solving in the field of software engineering, providing valuable information and guidance to developers and engineering teams. Therefore, in classification tasks related to the field of software engineering, the use of automated methods to classify data has broad prospects and potential benefits.

[0004] However, despite the success of automated classification methods in classification tasks related to software engineering, there are still some unexpected classification errors. By analyzing these misclassification cases in the field of software engineering, it can be found that when two categories have similar features at the class level, the classification model can easily misclassify samples of these categories, resulting in a high probability of misclassification and reduced classification accuracy. This behavior of the classification model is called a confusing error in this field, the corresponding two categories are called confusing categories, and the similar features of the two categories are called confusing features. The existence of confusing features makes it easy for samples in the confusing category to be misclassified by the classification model.

[0005] In order to solve the classification error problem of classification tasks related to software engineering, the existing technology mainly adopts automatic classification methods based on classification models. There are two types of automatic classification methods based on classification models: the first type of method is to improve the robustness of the model based on adversarial training to reduce classification errors. For example, Papernot et al. proposed defensive distillation in "Distillation as a Defense to Adversaria Perturbations Against Deep Neural Networks" published in SP2016. This method effectively resists the attack of adversarial samples (see the document "Explaining and harnessing adversarial examples", 2014) without changing the model structure and affecting the model accuracy as little as possible. In "Towards Deep Learning Models Resistant to Adversarial Attacks" published at ICLR 2018, Gu et al. proposed a deep contraction network (DCN), a model of an end-to-end training process that uses a layered contraction penalty and a smoothness penalty term to improve the robustness of deep neural networks to adversarial samples by smoothing the model output.

[0006] The second type of method uses data augmentation and fine-tuning techniques to improve classification accuracy. For example, Mia et al. proposed and implemented three data augmentation techniques in "Data Augmentation for Improving Emotion Recognition in Software Engineering Communication" published at ASE2022, namely unconstrained, vocabulary, and polarity, to improve the accuracy of the classification system. The authors conducted experiments based on the current optimal sentiment classification model SEntiMoji (called the SEntiMoji method) (see the document "SEntiMoji: An Emoji-Powered Learning Approach for Sentiment Analysis in Software Engineering", 2019, SEntiMoji: An Emoji-Powered Learning Approach for Sentiment Analysis in Software Engineering). The results show that the data enhanced by the polarity strategy performs best on the SEntiMoji method, and the classification accuracy is improved. Devlin et al. proposed and implemented a pre-training model in "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" published in 2018. The authors pre-trained the BERT model on a large-scale corpus to give it a powerful representation capability (this method is referred to as the BERT method). Subsequently, only a small amount of labeled data is needed for fine-tuning to achieve excellent performance in various downstream tasks (such as classification tasks). This fine-tuning process enables the model to adapt to different downstream tasks and achieve better classification results.

[0007] However, existing technologies still have defects in solving the problem of easily confused classification in classification tasks. For example, when using the SEntiMoji model to classify a typical software engineering dataset, the model misclassifies the category "fear" as "anger" with a probability of up to 32%, and misclassifies the category "love" as "happiness" with a probability of up to 36%, which is much higher than the model's misclassification rate for other categories. When using the BERT model to classify a vulnerability dataset, the model misclassifies the category "input validation error" as "privilege escalation" with a probability of up to 35%, which is much higher than the model's misclassification rate for other categories.

[0008] Therefore, how to make full use of the different features learned by the model from easily confused category data and provide a more effective classification method to improve classification accuracy is a hot issue that technicians in this field are discussing. Summary of the invention

[0009] The technical problem to be solved by the present invention is to provide a method for classifying easily confused categories based on a two-stage model, in view of the fact that easily confused categories are easily confused when the existing automated classification method based on the classification model is used to classify software engineering related data, resulting in low classification accuracy of easily confused categories. The method improves the classification accuracy of the model for easily confused categories by identifying easily confused categories and reclassifying them.

[0010] In order to solve the above technical problems, the technical solution of the present invention is: first, a two-stage classification system consisting of a preprocessing module, an easily confused category recognition module, a first-stage classification module, a boundary sample screening module and a second-stage classification module is constructed. A training set is constructed using the Polarity and MSR data sets. The second-stage classification module is trained in a supervised manner using the training set to obtain a trained second-stage classification module. Then, the trained two-stage classification system is used to preprocess and classify the target sample (including text or source code) input by the user, and the easily confused category binary is obtained through the easily confused category recognition module, and the boundary sample is identified through the boundary sample screening module. If it is a boundary sample, the second-stage classification module is used for reclassification, and finally the predicted label sequence, that is, the classification result, is obtained, that is, the category to which the input data belongs, to achieve the classification of the target data.

[0011] The present invention comprises the following steps:

[0012] The first step is to build a two-stage classification system. The two-stage classification system consists of a preprocessing module, a first-stage classification module, a confusing category recognition module, a boundary sample screening module, and a second-stage classification module.

[0013] The preprocessing module is connected to the first stage classification module and the second stage classification module. During training, the first training set T or the second training set RR is read, T = {T 1 ,T 2 ,···T m ,···,T M}, where T m represents the mth text sequence in T (where 1≤m≤M, M is the total number of texts in T), Indicates T m RR = {R 1 ,R 2 ,···R n ,···,R N}, where R n represents the nth source code sequence in RR (where 1≤n≤N, N is the total number of source codes in RR); Represents R n The i-th word (the letters between two spaces in the source code constitute a word, and the source code also consists of I words, I is T m The total number of words in T) (where 1≤i≤I). T consists of M texts, where a word refers to a basic unit in the text. For example, in the text “I am happy.”, “I”, “am”, “happy”, and “.” are words. The training set RR consists of N source codes. A word refers to a basic unit in the source code segment separated by spaces. For example, in the source code “if(num>i){num++;}”, “if”, “(”, “num”, “>”, “i”, “)”, “{”, “num”, “++”, “;”, and “}” are words. The word embedding method is used to map each word in the training set to a word-level encoding to obtain a set of word-level encoding sequences for the training set. The preprocessing module sends the set of word-level encoding sequences for the training set to the second-stage classification module.

[0014] When classifying the text or source code input by the user, the preprocessing module receives the text or source code sequence X input by the user, X = {x1, x2, ..., x j ,...,x J}(where x j represents the jth word in X, 1≤j≤J, J is the number of words in X), and uses the word embedding method to map each word to a word-level code, and obtains the user word-level code sequence set G(X), G(X) = {G(x1), G(x2), ..., G(x j ),...,G(x J )}(where G(x j ) represents x j The word-level encoding after mapping by the word embedding method), the preprocessing module sends G(X) to the first-stage classification module and the second-stage classification module.

[0015] When it is necessary to obtain confusable category binary tuples, the preprocessing module reads the test set, maps each word in the test set to a word-level code using the word embedding method, and obtains a test set word-level code sequence set. The preprocessing module sends the test set word-level code sequence set to the first-stage classification module.

[0016] The first-stage classification module is connected to the preprocessing module, the easily confused category identification module, and the boundary sample screening module, and receives the category table input by the user. The category table consists of two fields: category name and category subscript. The category name corresponding to category subscript 1 is A, the category name corresponding to category subscript 2 is B, and so on. For example, in the category table of the software engineering sentiment classification task input by the user, the category name corresponding to category subscript 1 is "anger", the category name corresponding to category subscript 2 is "love", and the category name corresponding to category subscript 6 is "fear". The first-stage classification module receives the test set word-level coding sequence set and G(X) from the preprocessing module, classifies the test set word-level coding sequence set, obtains the confusion matrix, and sends the confusion matrix and category table to the easily confused category identification module. The first-stage classification module classifies G(X) and obtains the probability set P of each category in the category table input by the user corresponding to X, P={p1,p2,...,p u ,...,p U}(where p uIt indicates that the first-stage classification module predicts the probability that G(X) belongs to the category corresponding to the subscript u, 1≤u≤U), takes the category corresponding to the highest probability as the category cat1 of X, and sends P and cat1 to the boundary sample screening module. If the object X to be classified is text (i.e., X received by the preprocessing module from the user is text), the first-stage classification module is any one of ML (see the literature “Machine learning”, 1997, machine learning), RNN (see the literature “Recurrent neural network regularization”, 2014, recurrent neural network), LSTM (see the literature “Long short-term memory”, 1997, long short-term memory), BERT (see the literature “Bert: Pre-training of deep bidirectional transformers for language understanding”, 2018, Bert: A pre-trained model for deep bidirectional transformation for language understanding); if the object X to be classified is source code (i.e., X received by the preprocessing module from the user is source code), the first-stage classification module is ML (see the literature “Machine learning”, 1997, machine learning), GGNN (see the literature “Gated graph sequence neural networks”, 2015, gated graph sequence neural networks), LSTM (see the literature “Long short-term memory”, 1997, long short-term memory), CodeBERT (see the literature “Codebert: A pre-trained model for deep bidirectional transformation for language understanding”). model for programming and natural languages", 2020, Codebert: A pre-trained model for programming and natural languages).

[0017] The easily confused category identification module is connected to the first-stage classification module and the boundary sample screening module. It receives the category table and confusion matrix from the first-stage classification module, arranges all the non-diagonal values ​​in the confusion matrix from small to large, and obtains the third quartile Q3. Calculate twice Q3 to obtain the threshold for judging the easily confused category. Compare all the non-diagonal elements in the confusion matrix with the easily confused category threshold to obtain the easily confused category binary sequence. Send the easily confused category binary sequence and the category table to the boundary sample screening module.

[0018] The boundary sample screening module is connected to the first-stage classification module, the easily confused category recognition module, and the second-stage classification module. It receives P and cat1 from the first-stage classification module, receives the easily confused category binary sequence and category table from the easily confused category recognition module, sets the reclassification threshold to ω, and classifies category cat1 and

[0019] The easily confused category binary sequences are compared. If category cat1 does not belong to

[0020] If the category of X is a binary in the easily confused category binary sequence, cat1 is taken as the category of X; if category cat1 belongs to a binary in the easily confused category binary sequence, the probability interpolation Δp is calculated. If Δp>ω, cat1 is taken as the category of X; if Δp<ω, the signal flag that X needs to be reclassified is set to 1, and the flag is sent to the second stage classification module.

[0021] The second-stage classification module consists of a feedforward neural network layer and a linear classification layer. If the object X to be classified is text, when the first-stage classification module is any one of ML, RNN, and LSTM, the second-stage classification module is BERT; when the first-stage classification module is BERT, the second-stage classification module is RoBERTa. If the object X to be classified is source code, when the first-stage classification module is any one of ML, GGNN, and LSTM, the second-stage classification module is CodeBERT; when the first-stage classification module is CodeBERT, the second-stage classification module is GraphCodeBERT (see the document "Graphcodebert: Pre-training code representations with data flow", 2020, Graphcodebert: Pre-training code representations with data flow).

[0022] The feedforward neural network layer is connected to the preprocessing module, the linear classification layer, and the boundary sample screening module. During training, the feedforward neural network layer receives the training set word-level coding sequence set from the preprocessing module, performs nonlinear mapping on each word-level coding in the training set word-level coding sequence set, obtains the nonlinearly mapped representation Attn of the training set word-level coding sequence set, and sends Attn to the linear classification layer. When classifying the text or source code input by the user, the feedforward neural network layer receives the flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is nonlinearly mapped to obtain the nonlinearly mapped representation Attn′ of G(X), and Attn′ is sent to the linear classification layer.

[0023] The linear classification layer is connected to the feedforward neural network layer. During training, the linear classification layer receives Attn from the feedforward neural network layer, performs linear mapping on Attn, obtains the first prediction vector, i.e., logits, performs Softmax operation on logits, and obtains the probability set P′ of each category in the category table corresponding to each sample in the training set.

[0024] P′={p′1,p′2,...,p′ u ,...,p′ U}(where p′ u Represents the second stage classification model prediction T m The probability of belonging to the category corresponding to the subscript u, 1≤u≤U), the category corresponding to the highest probability is taken as the category cat2 of each sample in the training set, and the cross loss entropy function is used to calculate the loss Loss for back propagation to update the model parameters.

[0025] When classifying the text or source code input by the user, the linear classification layer receives Attn′ from the feedforward neural network layer, performs linear mapping on Attn′, obtains the second prediction vector logits′, performs Softmax operation on logits′, and obtains the second probability set PP′ for each category in the category table corresponding to G(X).

[0026] PP′={pp′1,pp′2,...,pp′ u ,...,pp′ U}(where pp′ u It represents the probability that the second-stage classification model predicts that X belongs to the category corresponding to the subscript u, 1≤u≤U). The category corresponding to the highest probability is taken as the category cat2′ of X, and the category cat2′ is output.

[0027] The second step is to prepare the dataset for training the two-stage classification system by:

[0028] 2.1 Use the text data in the Polarity dataset to construct the first training set and the first test set for the software engineering sentiment classification task. Put AA (AA is a positive integer, preferably 15436) samples in the Polarity dataset (taken from https: / / anonymous.4open.science / r / SE-Emotion-Study-0141 / , 2022) into the first training set, and BB (BB is a positive integer, preferably 349) samples into the first test set. Each sample in the Polarity dataset contains a text and its corresponding actual label. The text set in the first training set constitutes the first training set set T, and the actual label sequence set in the first training set constitutes the first training set actual label sequence set T # ; Let T = {T 1,T 2 ,···T m ,···,T M}, where T m represents the mth text sequence in T (where 1≤m≤M, M is the total number of texts in T); Indicates T m The text set in the first test set constitutes the first test set V, V = {V 1 ,V 2 ,···V r ,···,V R}, where V r represents the rth text in V, where 1≤r≤R, R is the total number of texts in V, in Indicates V r The qth word in V, 1≤q≤Q, Q is r The actual label sequence set in the first test set samples constitutes the actual label sequence set V of the first test set # If you need to classify text, use T to train a two-stage classification system, T # It is used to calculate the training loss Loss and calibrate the model when training the two-stage classification system; V is used to test the two-stage classification system, V # Used to compare with the test output labels to measure the performance of the two-stage classification system being tested.

[0029] 2.2 Use the source code data in the MSR dataset to construct the second training set and the second test set for the software engineering sentiment classification task. Put CC (CC is a positive integer, preferably 6670) samples in the MSR dataset (taken from https: / / doi.org / 10.6084 / m9.figshare.20791240, 2023) into the training set, and DD (DD is a positive integer, preferably 845) samples into the second test set. Each sample in the MSR dataset contains a source code and its corresponding actual label. The source code set in the second training set constitutes the second training set set RR, and the actual label sequence set in the second training set constitutes the second training set actual label sequence set RR. # ; Let RR = {R 1 ,R 2 ,···R n ,···,R N}, where R n represents the nth source code sequence in RR (where 1≤n≤N, N is the total number of source codes in RR); Represents Rn The source code set in the second test set constitutes the second test set S, S = {S 1 ,S 2 ,···S o ,···,S O}, where S o represents the o-th text in S, where 1≤o≤O, O is the total number of source codes in S, in Indicates S o The qqth word in S, 1≤qq≤QQ, QQ is S o The actual label sequence set in the second test set samples constitutes the actual label sequence set S of the second test set # If source code classification is required, a two-stage classification system is trained using RR. # It is used to calculate the training loss Loss and calibrate the model when training the two-stage classification system; S is used to test the two-stage classification system, S # Used to compare with the test output labels to measure the performance of the two-stage classification system being tested.

[0030] In the third step, if you need to classify text data, go to the fourth step; if you need to classify source code data, go to the fifth step.

[0031] In the fourth step, the preprocessing module uses a preprocessing method to map the text data in T to the first training set word-level coding sequence set G(T), and maps the text data in V to the test set word-level coding sequence set G(V), the method is:

[0032] 4.1 Use the word-level mapping method to map the M text data in T to the first training set word-level encoding sequence set G(T), the method is:

[0033] 4.1.1 Let variable m=1.

[0034] 4.1.2 Read the mth text T in T m , using word-level encoding method to m of Perform word mapping and get T m The word-level encoding sequence G(T m ):

[0035] 4.1.2.1 Let variable i=1.

[0036] 4.1.2.2 T m The i-th word in is mapped to the word-level encoding (See the literature “Attention is all you need.”, 2017, Attention is all you need).

[0037] 4.1.2.3 Let i=i+1. If i≤I, go to 4.1.2.2. If i>I, then T m The I words in T have been mapped to word-level encodings, and the word-level encoding sequence of the mth text data in T is obtained. Indicates T m For the i-th word, go to 4.1.3.

[0038] 4.1.3 Let m = m + 1. If m ≤ M, go to 4.1.2. If m > M, it means that the M text data of T have been mapped to word-level coding sequences, and G(T) is obtained. The word-level coding sequence set G(T) = {G(T 1 ),G(T 2 ),...,G(T m ),...,G(T M )},G(T m ) represents the word-level encoding sequence of the mth text data in T.

[0039] 4.2 Use the word-level mapping method described in 4.1 to map the R text data in V into the first test set word-level encoding set G(V).

[0040] 4.3 Send G(T) to the second-stage classification module, send G(V) to the first-stage classification module, and go to step 6.

[0041] In the fifth step, the preprocessing module adopts the preprocessing method described in the fourth step to map the source code data in RR to the second training set word-level coding sequence set G(RR), and maps the source code data in S to the second test set word-level coding sequence set G(S), sends G(RR) to the second-stage classification module, and sends G(S) to the first-stage classification module, and then goes to the seventh step.

[0042] In the sixth step, the first stage classification module receives the category table input by the user, receives G(V) from the preprocessing module, classifies G(V) using a classification method, and obtains the first confusion matrix CM, which is:

[0043] 6.1 The first stage classification module receives the category table input by the user and receives G(V) from the preprocessing module. The category table consists of two fields: category name and category subscript. U is the total number of category names in the category table.

[0044] 6.2 The first stage classification module classifies G(V) and obtains the category set pred.

[0045] 6.3 The first stage classification module uses the confusion_matrix function (from the scikit-learn library version 1.0.2) to obtain the first confusion matrix CM, CM = {p 11 ,p 12 ,...p uu′ ,...,p UU ,}(where p uu′ It represents the probability that the first-stage classification module predicts the category corresponding to subscript u as the category corresponding to subscript u′, 1≤u≤U, 1≤u′≤U).

[0046] 6.4 The first-stage classification module sends the CM and category table to the easily confused category identification module and goes to step 8.

[0047] In the seventh step, the first-stage classification module receives the category table input by the user, receives G(S) from the preprocessing module, uses the classification method described in the sixth step to classify G(S), obtains the second confusion matrix CM′, sends CM′ and the category table to the easily confused category recognition module, and then goes to the ninth step.

[0048] In the eighth step, the easily confused category recognition module uses the first confusion matrix CM to obtain the first easily confused category binary sequence L by using the recognition method, the method is:

[0049] 8.1 Initialize the first easily confused category binary sequence L to be empty.

[0050] 8.2 Arrange all non-diagonal values ​​in the first confusion matrix CM from small to large.

[0051] 8.3 Calculate the third quartile Q3 of the arranged CM.

[0052] 8.4 Let the confusing category threshold δ = 2Q3.

[0053] 8.5 Let variable u=1.

[0054] 8.6 Let variable u′=1.

[0055] 8.7 Comparison of p uu′ and δ. If p uu′ >δ, then the category corresponding to the subscript u and the category corresponding to the subscript u′ are a set of easily confused category tuples, and the tuple (category corresponding to u, category corresponding to u′) is added to L.

[0056] 8.8 Let u′=u′+1. If u′≤U, go to 8.7. If u′>U, it means that a row in CM has been traversed, so go to 8.9.

[0057] 8.9 Let u=u+1. If u≤U, go to 8.7. If u>U, it means that CM has been completely traversed and the first easily confused category binary sequence L is obtained, L={l1,l2,...,l e ,...,l E}(where l e is a pair of easily confused category binary tuples in the form of (A, B), indicating that category A (the category corresponding to u) and category B (the category corresponding to u′) are easily confused by the model, 1≤e≤E, E is the number of binary tuples identified by the easily confused category identification module, that is, the number of binary tuples in L) Send L to the boundary sample screening module and go to step 10.

[0058] In the ninth step, the easily confused category identification module adopts the identification method described in the eighth step, uses the second confusion matrix CM′ to obtain the second easily confused category binary sequence LL, sends LL to the boundary sample screening module, and then goes to the eleventh step.

[0059] The tenth step is to use the training method to train the two-stage classification system in a supervised manner using T to obtain the network weight parameters of the feedforward neural network layer and the linear classification layer of the second-stage classification module. The method is:

[0060] 10.1 Initialize the weight parameters and set the parameter set W of the feedforward neural network layer F Initialize the elements in to random numbers in [0,1]; set the parameter set W of the linear classification layer Y The elements in are initialized to random numbers in [0,1].

[0061] 10.2 Set the network training hyperparameters, set the word embedding size to 768, and the network model learning rate to 5×10 -5 , set the weight decay rate (weight_decay) value to 0.01, the number of heads (head) to 12, the warmup learning rate (warmup) to 0.05, the batch size (batch_size) to 16, and the encoding dropout rate (dropout) to 0.5.

[0062] 10.3 The second stage classification module receives G(T) from the preprocessing module and uses supervised iterative training. Each round of training obtains the first predicted label sequence T′ and calculates T′ and T # The distribution gap is obtained, the loss value is minimized and the network parameters of the feedforward neural network layer and the linear classification layer are updated until the number of iterations is met, and the weight parameters of the feedforward neural network layer and the linear classification layer are obtained. The method is as follows:

[0063] 10.3.1 Initialize the training iteration parameter epoch = 1; initialize the iteration threshold θ as a positive integer, preferably θ = 20.

[0064] 10.3.2 Let variable m=1.

[0065] 10.3.3 The feedforward neural network layer uses a nonlinear mapping method (see the paper "Attention is all you need.", 2017) to map G(T m ) is nonlinearly mapped to obtain G(T m ) is represented by the nonlinear mapping Attn, and Attn is sent to the linear classification layer as the sentence-level encoding representation.

[0066] 10.3.4 Linear classification layer maps Attn to the first prediction vector logits of the category number dimension, performs softmax operation on logits, and obtains T m Corresponding to the probability of each category in the category table input by the user, the category with the highest probability is taken as T m Category cat2.

[0067] 10.3.5 The linear classification layer uses the cross-loss entropy function to calculate T' and T # The distribution gap between them is used to obtain the loss value Loss. The Adam optimization algorithm (see the literature "A method for stochastic optimization", 2015, a stochastic optimization method) is used to minimize the Loss. The weight parameters of the feedforward neural network layer and the linear mapping layer are updated once through back propagation to obtain the updated W F and W Y .

[0068] 10.3.6 Let m = m + 1. If m ≤ M, go to 10.3.3. If m>M, it means that all M data samples of T have participated in the training and one iteration of training has been completed. Go to 10.3.7.

[0069] 10.3.7 Let epoch = epoch + 1. If epoch ≤ iteration threshold θ, go to 10.3.2; otherwise, it means that the training meets the iteration number requirement, the training is completed, and the weight parameters of the feedforward neural network layer and the linear mapping layer in the second-stage classification module are obtained, and go to step 12.

[0070] In the eleventh step, the training method described in the tenth step is adopted, and the second training set RR is used to perform supervised training on the two-stage classification system to obtain the network weight parameters of the feedforward neural network layer and the linear classification layer in the second-stage classification module, and then go to the twelfth step.

[0071] In the twelfth step, the weight parameters obtained through training are assigned to the second-stage classification module, that is, the trained W FAs the network weight parameter of the feedforward neural network layer of the second-stage classification module, the trained W Y As the weight parameter of the linear classification layer of the second-stage classification module, a trained two-stage classification system is obtained.

[0072] Step 13: If you need to classify text data, go to step 14. If you need to classify source code data, go to step 15.

[0073] In the fourteenth step, the trained two-stage classification system classifies the text X input by the user to obtain the category to which the text X belongs. The method is:

[0074] 14.1 The preprocessing module receives the text X input by the user, X={x1,x2,···,x i ,···,x I}(where x i represents the i-th word in the input source code), use the word-level encoding method described in step 4.1.2 to encode x1, x2, ···, x in X. i ,···,x I Perform word mapping to obtain the word-level encoding sequence set G(X) of X, G(X) = {G(x1), G(x2), ··G(x i ),···,G(x I )},G(x i ) is x i The word-level encoding is performed and G(X) is sent to the first-stage classification module and the second-stage classification module.

[0075] 14.2 The first stage classification module receives G(X) from the preprocessing module, classifies G(X), and obtains the probability set P of each category in the category table input by the user corresponding to X, P = {p1, p2, ..., p u ,...,p U}(where p u It represents the probability that the first-stage classification model predicts that X belongs to the category corresponding to the subscript u, 1≤u≤U). The category corresponding to the highest probability is taken as the category cat1 of X, and the obtained probability set P and category cat1 are sent to the boundary sample screening module.

[0076] 14.3 The boundary sample screening module receives P and cat1 from the first classification module, receives the easily confused category binary sequence L from the easily confused category identification module, and determines whether X needs to be reclassified using the second stage classification module based on L, P and cat1. The method is:

[0077] 14.3.1 Let e ​​= 1, and take the tuple l from L e .

[0078] 14.3.2 If cat1 does not belong to the tuple l e In the category, go to 14.3.3; if category cat1 belongs to the tuple l e The categories in the table indicate that u corresponds to l e The subscript of the first category in , u′ corresponds to l e For the subscript of the second category, go to 14.3.5.

[0079] 14.3.3 Let e ​​= e + 1. If e ≤ E, go to 14.3.2. If e > E, it means that cat1 has been compared with every pair in L, and cat1 does not belong to any pair of pairs in L. Go to 14.3.4.

[0080] 14.3.4 Take cat1 as the predicted category of X, output cat1, and go to step 16.

[0081] 14.3.5 Calculate the probability interpolation Δp = |p u -p u′ |.

[0082] 14.3.6 Let the reclassification threshold ω be a decimal between 0 and 1 (preferably ω = 0.2). If Δp>ω, take cat1 as the category of X, output cat1, and go to step 16; if Δp<ω, set the signal flag that needs to reclassify G(X) to 1 and send it to the second stage classification module, and go to 14.4.

[0083] 14.4 The second stage classification module receives the signal flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is classified and the category corresponding to the highest probability is taken as the category cat2 of X. The method is:

[0084] 14.4.1 The feedforward neural network layer receives the word-level code G(X) from the preprocessing module, uses the nonlinear mapping method to perform nonlinear mapping on G(X), obtains the nonlinear mapped representation Attn′ of G(X), and sends Attn′ as the sentence-level code representation to the linear classification layer.

[0085] 14.4.2 The linear classification layer maps Attn′ to the second prediction vector logits′ in the dimension of the number of categories, performs a softmax operation on logits′, obtains the probability of each category in the category table corresponding to X, takes the category with the highest probability as the predicted label cat2 of X, and goes to step 16.

[0086] Step 15: The trained two-stage classification system classifies the source code X input by the user to obtain the category to which the source code X belongs. The method is:

[0087] 15.1 The preprocessing module receives the source code X input by the user, X = {x1, x2, ···, x i ,···,x I}(where x i represents the i-th word in the input source code), use the word-level encoding method described in step 4.1.2 to encode x1, x2, ···, x in X. i ,···,x I Perform word mapping to obtain the word-level encoding sequence set G(X) of X, G(X) = {G(x1), G(x2), ··G(x i ),···,G(x I )},G(x i ) is x i The word-level encoding is performed and G(X) is sent to the first-stage classification module and the second-stage classification module.

[0088] 15.2 The first stage classification module receives the word-level encoding G(X) of X from the preprocessing module, classifies G(X), and obtains the probability set P of each category in the category table corresponding to X, P = {p1, p2, ..., p u ,...,p U}(where p u It represents the probability that the first-stage classification model predicts that X belongs to the category corresponding to the subscript u, 1≤u≤U). The category corresponding to the highest probability is taken as the category cat1 of X, and the obtained probability set P and category cat1 are sent to the boundary sample screening module.

[0089] 15.3 The boundary sample screening module receives P and cat1 from the first classification module, and receives the second easily confused category binary sequence LL from the easily confused category identification module, and determines whether X needs to be reclassified using the second stage classification module through LL, P and cat1. The method is:

[0090] 15.3.1 Let e ​​= 1, and take the tuple ll from LL e .

[0091] 15.3.2 If cat1 does not belong to the tuple l e In the category, go to 15.3.3; if category cat1 belongs to the tuple l e The categories in the table indicate that u corresponds to l e The subscript of the first category in , u′ corresponds to l e For the subscript of the second category, go to 15.3.5.

[0092] 15.3.3 Let e=e+1. If e≤E, go to 15.3.2. If e>E, it means that cat1 has been compared with every pair in L, and cat1 does not belong to any pair of pairs in L. Go to 15.3.4.

[0093] 15.3.4 Take cat1 as the predicted category of X, output cat1, and go to step 16.

[0094] 15.3.5 Calculate the probability interpolation Δp = |p u -p u′ |.

[0095] 15.3.6 Let the reclassification threshold ω be a decimal between 0 and 1 (preferably ω = 0.2). If Δp>ω, take cat1 as the category of X, output cat1, and go to step 16; if Δp<ω, set the signal flag that needs to reclassify G(X) to 1 and send it to the second stage classification module, and go to 15.4.

[0096] 15.4 The second stage classification module receives the signal flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is classified to obtain the predicted class cat2 of X. The method is:

[0097] 15.4.1 The feedforward neural network layer receives the word-level code G(X) from the preprocessing module, uses the nonlinear mapping method to perform nonlinear mapping on G(X), obtains the nonlinear mapped representation Attn′ of G(X), and sends Attn′ as the sentence-level code representation to the linear classification layer.

[0098] 15.4.2 The linear classification layer maps Attn′ to the second prediction vector logits′ in the dimension of the number of categories, performs a softmax operation on logits′, obtains the probability of each category in the category table corresponding to X, takes the category with the highest probability as the predicted label cat2 of X, and goes to step 16.

[0099] Step 16. End.

[0100] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0101] 1. The present invention can effectively identify easily confused categories in typical software engineering-related classification tasks. For text classification tasks (such as software engineering emotion recognition tasks), the present invention identifies two pairs of easily confused categories, namely (anger, fear) and (love, happiness). These two pairs of easily confused categories have an important impact on public opinion analysis, emotion monitoring and other fields. Correctly classifying anger and fear, love and happiness greatly improves the accuracy of emotion classification. For source code classification tasks (such as vulnerability classification tasks), the present invention identifies a pair of easily confused categories, namely (input verification error, privilege escalation). (Input verification error, privilege escalation) This pair of easily confused categories seriously threatens the security and reliability of the classification software. Correctly classifying input verification errors and privilege escalation can greatly improve the security and reliability of the classification system.

[0102] 2. The present invention can effectively distinguish easily confused categories and improve the F1 value of classification. The F1 value is the weighted average of precision and recall, and is a comprehensive evaluation index used to measure the performance of the classification system. For text classification tasks (taking the software engineering emotion recognition task as an example), the present invention has a relative improvement of 5.08% in the F1 value when classifying the anger class; the present invention has a relative improvement of 10.2% in the F1 value when classifying the love class; the present invention has a relative improvement of 2.42% in the F1 value when classifying the happiness class; the present invention has a relative improvement of 19.8% in the F1 value when classifying the fear class; the present invention makes the average F1 value of all categories in the software engineering emotion recognition task category table relatively improved by 5.7%. For source code classification tasks (taking vulnerability classification tasks as an example), the present invention has a relative improvement of 8.62% in the F1 value when classifying input verification error classes; the present invention has a relative improvement of 2.46% in the F1 value when classifying the privilege escalation class; the present invention makes the average F1 value of all categories in the vulnerability classification task category table relatively improved by 1.91%. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] Figure 1 It is the overall flow chart of the present invention;

[0104] Figure 2 It is the logical framework diagram of the first step of the present invention to construct a two-stage classification system.

[0105] Figure 3 is an example of a category table for user input, a category table for a software engineering sentiment recognition task. DETAILED DESCRIPTION

[0106] The present invention will be described below in conjunction with the accompanying drawings and Example 1. Figure 1 As shown, embodiment 1 of the present invention includes the following steps:

[0107] The first step is to build a two-stage classification system. Figure 2 As shown, it consists of a preprocessing module, a first-stage classification module, an easily confused category recognition module, a boundary sample screening module, and a second-stage classification module.

[0108] The preprocessing module is connected to the first stage classification module and the second stage classification module. During training, the first training set T or the second training set RR is read, T = {T 1 ,T 2 ,···T m ,···,T M}, where T m represents the mth text sequence in T (where 1≤m≤M, M is the total number of texts in T), Indicates T m RR = {R 1 ,R 2 ,···R n ,···,R N}, where R n represents the nth source code sequence in RR (where 1≤n≤N, N is the total number of source codes in RR); Represents R n The i-th word (the letters between two spaces in the source code constitute a word, and the source code also consists of I words, I is T m The total number of words in T) (where 1≤i≤I). T consists of M texts, where a word refers to a basic unit in the text. For example, in the text “I am happy.”, “I”, “am”, “happy”, and “.” are words. The training set RR consists of N source codes. A word refers to a basic unit in the source code segment separated by spaces. For example, in the source code “if(num>i){num++;}”, “if”, “(”, “num”, “>”, “i”, “)”, “{”, “num”, “++”, “;”, and “}” are words. The word embedding method is used to map each word in the training set to a word-level encoding to obtain a set of word-level encoding sequences for the training set. The preprocessing module sends the set of word-level encoding sequences for the training set to the second-stage classification module.

[0109] When classifying the text or source code input by the user, the preprocessing module receives the text or source code sequence X input by the user, X = {x1, x2, ..., x j ,...,x J}(where x jrepresents the jth word in X, 1≤j≤J, J is the number of words in X), and uses the word embedding method to map each word to a word-level code, and obtains the user word-level code sequence set G(X), G(X) = {G(x1), G(x2), ..., G(x j ),...,G(x J )}(where G(x j ) represents x j The word-level encoding after mapping by the word embedding method), the preprocessing module sends G(X) to the first-stage classification module and the second-stage classification module.

[0110] When it is necessary to obtain confusable category binary tuples, the preprocessing module reads the test set, maps each word in the test set to a word-level code using the word embedding method, and obtains a test set word-level code sequence set. The preprocessing module sends the test set word-level code sequence set to the first-stage classification module.

[0111] The first stage classification module is connected with the preprocessing module, the confusing category identification module, and the boundary sample screening module, and receives the category table input by the user. The category table consists of two fields: category name and category subscript, such as Figure 3 As shown, in the category table of the software engineering sentiment classification task input by the user, the category name corresponding to category subscript 1 is "anger", the category name corresponding to category subscript 2 is "love", and the category name corresponding to category subscript 6 is "fear". The first-stage classification module receives the test set word-level coding sequence set and G(X) from the preprocessing module, classifies the test set word-level coding sequence set, obtains the confusion matrix, and sends the confusion matrix and category table to the easily confused category identification module. The first-stage classification module classifies G(X) and obtains the probability set P of each category in the category table input by the user corresponding to X, P = {p1, p2, ..., p u ,...,p U}(where p uIt indicates that the first-stage classification module predicts the probability that G(X) belongs to the category corresponding to the subscript u, 1≤u≤U), takes the category corresponding to the highest probability as the category cat1 of X, and sends P and cat1 to the boundary sample screening module. If the object X to be classified is text (i.e., X received by the preprocessing module from the user is text), the first-stage classification module is any one of ML (see the literature “Machine learning”, 1997, machine learning), RNN (see the literature “Recurrent neural network regularization”, 2014, recurrent neural network), LSTM (see the literature “Long short-term memory”, 1997, long short-term memory), and BERT (see the literature “Bert: Pre-training of deep bidirectional transformers for language understanding”, 2018, Bert: A deep bidirectional transformer pre-training model for language understanding); if the object X to be classified is source code (i.e., X received by the preprocessing module from the user is source code), the first-stage classification module is ML (see the literature “Machine learning”, 1997, machine learning), GGNN (see the literature “Gated graph sequence neural networks”, 2015, gated graph sequence neural networks), LSTM (see the literature “Long short-term memory”, 1997, long short-term memory), CodeBERT (see the literature “Codebert: A Any of "Codebert: pre-trained model for programming and natural languages", 2020.

[0112] The easily confused category identification module is connected to the first-stage classification module and the boundary sample screening module. It receives the category table and confusion matrix from the first-stage classification module, arranges all the non-diagonal values ​​in the confusion matrix from small to large, and obtains the third quartile Q3. Calculate twice Q3 to obtain the threshold for judging the easily confused category. Compare all the non-diagonal elements in the confusion matrix with the easily confused category threshold to obtain the easily confused category binary sequence. Send the easily confused category binary sequence and the category table to the boundary sample screening module.

[0113] The boundary sample screening module is connected to the first-stage classification module, the easily confused category recognition module, and the second-stage classification module. It receives P and cat1 from the first-stage classification module, receives the easily confused category binary sequence and category table from the easily confused category recognition module, sets the reclassification threshold to ω, and classifies category cat1 and

[0114] The easily confused category binary sequences are compared. If category cat1 does not belong to

[0115] If the category of X is a binary in the easily confused category binary sequence, cat1 is taken as the category of X; if category cat1 belongs to a binary in the easily confused category binary sequence, the probability interpolation Δp is calculated. If Δp>ω, cat1 is taken as the category of X; if Δp<ω, the signal flag that X needs to be reclassified is set to 1, and the flag is sent to the second stage classification module.

[0116] The second-stage classification module consists of a feedforward neural network layer and a linear classification layer. If the object X to be classified is text, when the first-stage classification module is any one of ML, RNN, and LSTM, the second-stage classification module is BERT; when the first-stage classification module is BERT, the second-stage classification module is RoBERTa. If the object X to be classified is source code, when the first-stage classification module is any one of ML, GGNN, and LSTM, the second-stage classification module is CodeBERT; when the first-stage classification module is CodeBERT, the second-stage classification module is GraphCodeBERT (see the document "Graphcodebert: Pre-training code representations with dataflow", 2020, Graphcodebert: Pre-training code representations with dataflow).

[0117] The feedforward neural network layer is connected to the preprocessing module, the linear classification layer, and the boundary sample screening module. During training, the feedforward neural network layer receives the training set word-level coding sequence set from the preprocessing module, performs nonlinear mapping on each word-level coding in the training set word-level coding sequence set, obtains the nonlinearly mapped representation Attn of the training set word-level coding sequence set, and sends Attn to the linear classification layer. When classifying the text or source code input by the user, the feedforward neural network layer receives the flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is nonlinearly mapped to obtain the nonlinearly mapped representation Attn′ of G(X), and Attn′ is sent to the linear classification layer.

[0118] The linear classification layer is connected to the feedforward neural network layer. During training, the linear classification layer receives Attn from the feedforward neural network layer, performs linear mapping on Attn, obtains the first prediction vector, i.e., logits, performs Softmax operation on logits, and obtains the probability set P′ of each category in the category table corresponding to each sample in the training set.

[0119] P′={p′1,p′2,...,p′ u ,...,p′ U}(where p′ u Represents the second stage classification model prediction T m The probability of belonging to the category corresponding to the subscript u, 1≤u≤U), the category corresponding to the highest probability is taken as the category cat2 of each sample in the training set, and the cross loss entropy function is used to calculate the loss Loss for back propagation to update the model parameters.

[0120] When classifying the text or source code input by the user, the linear classification layer receives Attn′ from the feedforward neural network layer, performs linear mapping on Attn′, obtains the second prediction vector logits′, performs Softmax operation on logits′, and obtains the second probability set PP′ for each category in the category table corresponding to G(X).

[0121] PP′={pp′1,pp′2,...,pp′ u ,...,pp′ U}(where pp′ u It represents the probability that the second-stage classification model predicts that X belongs to the category corresponding to the subscript u, 1≤u≤U). The category corresponding to the highest probability is taken as the category cat2′ of X, and the category cat2′ is output.

[0122] The second step is to prepare the dataset for training the two-stage classification system by:

[0123] 2.1 Use the text data in the Polarity dataset to construct the first training set and the first test set for the software engineering sentiment classification task. Put AA (15436 in this embodiment) samples in the Polarity dataset into the first training set, and BB (349 in this embodiment) samples into the first test set. Each sample in the Polarity dataset contains a text and its corresponding actual label. The text set in the first training set constitutes the first training set set T, and the actual label sequence set in the first training set constitutes the first training set actual label sequence set T # ; Let T = {T 1 ,T 2 ,···T m ,···,T M}, where T mrepresents the mth text sequence in T (where 1≤m≤M, M is the total number of texts in T); Indicates T m The text set in the first test set constitutes the first test set V, V = {V 1 ,V 2 ,···V r ,···,V R}, where V r represents the rth text in V, where 1≤r≤R, R is the total number of texts in V, in Indicates V r The qth word in V, 1≤q≤Q, Q is r The actual label sequence set in the first test set samples constitutes the actual label sequence set V of the first test set # If you need to classify text, use T to train a two-stage classification system, T # It is used to calculate the training loss Loss and calibrate the model when training the two-stage classification system; V is used to test the two-stage classification system, V # Used to compare with the test output labels to measure the performance of the two-stage classification system being tested.

[0124] 2.2 Use the source code data in the MSR dataset to construct the second training set and the second test set for the software engineering sentiment classification task. Put the CC (6670 in this embodiment) samples in the MSR dataset into the training set, and the DD (845 in this embodiment) samples into the second test set. Each sample in the MSR dataset contains a source code and its corresponding actual label. The source code set in the second training set constitutes the second training set set RR, and the actual label sequence set in the second training set constitutes the second training set actual label sequence set RR. # ; Let RR = {R 1 ,R 2 ,···R n ,···,R N}, where R n represents the nth source code sequence in RR (where 1≤n≤N, N is the total number of source codes in RR); Represents R n The source code set in the second test set constitutes the second test set S, S = {S 1 ,S 2 ,···S o ,···,S O}, where S orepresents the o-th text in S, where 1≤o≤O, O is the total number of source codes in S, in Indicates S o The qqth word in S, 1≤qq≤QQ, QQ is S o The actual label sequence set in the second test set samples constitutes the actual label sequence set S of the second test set # If source code classification is required, a two-stage classification system is trained using RR. # It is used to calculate the training loss Loss and calibrate the model when training the two-stage classification system; S is used to test the two-stage classification system, S # Used to compare with the test output labels to measure the performance of the two-stage classification system being tested.

[0125] In the third step, if you need to classify text data, go to the fourth step; if you need to classify source code data, go to the fifth step.

[0126] In the fourth step, the preprocessing module uses a preprocessing method to map the text data in T to the first training set word-level coding sequence set G(T), and maps the text data in V to the test set word-level coding sequence set G(V), the method is:

[0127] 4.1 Use the word-level mapping method to map the M text data in T to the first training set word-level encoding sequence set G(T), the method is:

[0128] 4.1.1 Let variable m=1.

[0129] 4.1.2 Read the mth text T in T m , using word-level encoding method to m of Perform word mapping and get T m The word-level encoding sequence G(T m ):

[0130] 4.1.2.1 Let variable i=1.

[0131] 4.1.2.2 T m The i-th word in is mapped to the word-level encoding (See the literature “Attention is all you need.”, 2017, Attention is all you need).

[0132] 4.1.2.3 Let i=i+1. If i≤I, go to 4.1.2.2. If i>I, then T mThe I words in T have been mapped to word-level encodings, and the word-level encoding sequence of the mth text data in T is obtained. Indicates T m For the i-th word, go to 4.1.3.

[0133] 4.1.3 Let m = m + 1. If m ≤ M, go to 4.1.2. If m > M, it means that the M text data of T have been mapped to word-level coding sequences, and G(T) is obtained. The word-level coding sequence set G(T) = {G(T 1 ),G(T 2 ),...,G(T m ),...,G(T M )},G(T m ) represents the word-level encoding sequence of the mth text data in T.

[0134] 4.2 Use the word-level mapping method described in 4.1 to map the R text data in V into the first test set word-level encoding set G(V).

[0135] 4.3 Send G(T) to the second-stage classification module, send G(V) to the first-stage classification module, and go to step 6.

[0136] In the fifth step, the preprocessing module adopts the preprocessing method described in the fourth step to map the source code data in RR to the second training set word-level coding sequence set G(RR), and maps the source code data in S to the second test set word-level coding sequence set G(S), sends G(RR) to the second-stage classification module, and sends G(S) to the first-stage classification module, and then goes to the seventh step.

[0137] In the sixth step, the first stage classification module receives the category table input by the user, receives G(V) from the preprocessing module, classifies G(V) using a classification method, and obtains the first confusion matrix CM, which is:

[0138] 6.1 The first stage classification module receives the category table input by the user and receives G(V) from the preprocessing module. The category table consists of two fields: category name and category subscript. U is the total number of category names in the category table.

[0139] 6.2 The first stage classification module classifies G(V) and obtains the category set pred.

[0140] 6.3 The first stage classification module uses the confusion_matrix function (from the scikit-learn library version 1.0.2) to obtain the first confusion matrix CM, CM = {p 11 ,p 12 ,...puu′ ,...,p UU ,}(where p uu′ It represents the probability that the first-stage classification module predicts the category corresponding to subscript u as the category corresponding to subscript u′, 1≤u≤U, 1≤u′≤U).

[0141] 6.4 The first-stage classification module sends the CM and category table to the easily confused category identification module and goes to step 8.

[0142] In the seventh step, the first-stage classification module receives the category table input by the user, receives G(S) from the preprocessing module, uses the classification method described in the sixth step to classify G(S), obtains the second confusion matrix CM′, sends CM′ and the category table to the easily confused category recognition module, and then goes to the ninth step.

[0143] In the eighth step, the recognition method is used to obtain the first easily confused category binary sequence L using the first confusion matrix CM, the method is:

[0144] 8.1 Initialize the first easily confused category binary sequence L to be empty.

[0145] 8.2 Arrange all non-diagonal values ​​in the first confusion matrix CM from small to large.

[0146] 8.3 Calculate the third quartile Q3 of the arranged CM.

[0147] 8.4 Let the confusing category threshold δ = 2Q3.

[0148] 8.5 Let variable u=1.

[0149] 8.6 Let variable u′=1.

[0150] 8.7 Comparison of p uu′ and δ. If p uu′ >δ, then the category corresponding to the subscript u and the category corresponding to the subscript u′ are a set of easily confused category tuples, and the tuple (category corresponding to u, category corresponding to u′) is added to L.

[0151] 8.8 Let u′=u′+1. If u′≤U, go to 8.7. If u′>U, it means that a row in CM has been traversed, so go to 8.9.

[0152] 8.9 Let u=u+1. If u≤U, go to 8.7. If u>U, it means that CM has been completely traversed and the first easily confused category binary sequence L is obtained, L={l1,l2,...,l e ,...,l E}(, where l eis a pair of easily confused category binary tuples in the form of (A, B), indicating that category A (the category corresponding to u) and category B (the category corresponding to u′) are easily confused by the model, 1≤e≤E, E is the number of binary tuples identified by the easily confused category identification module, that is, the number of binary tuples in L) Send L to the boundary sample screening module and go to step 10.

[0153] In the ninth step, the easily confused category identification module adopts the identification method described in the eighth step, uses the second confusion matrix CM′ to obtain the second easily confused category binary sequence LL, sends LL to the boundary sample screening module, and then goes to the eleventh step.

[0154] The tenth step is to use the training method to train the two-stage classification system in a supervised manner using T to obtain the network weight parameters of the feedforward neural network layer and the linear classification layer of the second-stage classification module. The method is:

[0155] 10.1 Initialize the weight parameters and set the parameter set W of the feedforward neural network layer F Initialize the elements in to random numbers in [0,1]; set the parameter set W of the linear classification layer Y The elements in are initialized to random numbers in [0,1].

[0156] 10.2 Set the network training hyperparameters, set the word embedding size to 768, and the network model learning rate to 5×10 -5 , set the weight decay rate (weight_decay) value to 0.01, the number of heads (head) to 12, the warmup learning rate (warmup) to 0.05, the batch size (batch_size) to 16, and the encoding dropout rate (dropout) to 0.5.

[0157] 10.3 The second stage classification module receives G(T) from the preprocessing module and uses supervised iterative training. Each round of training obtains the first predicted label sequence T′ and calculates T′ and T # The distribution gap is obtained, the loss value is minimized and the network parameters of the feedforward neural network layer and the linear classification layer are updated until the number of iterations is met, and the weight parameters of the feedforward neural network layer and the linear classification layer are obtained. The method is as follows:

[0158] 10.3.1 Initialize the training iteration parameter epoch = 1; initialize the iteration threshold θ = 20.

[0159] 10.3.2 Let variable m=1.

[0160] 10.3.3 The feedforward neural network layer uses a nonlinear mapping method (see the paper "Attention is all you need.", 2017) to map G(Tm ) is nonlinearly mapped to obtain G(T m ) is represented by the nonlinear mapping Attn, and Attn is sent to the linear classification layer as the sentence-level encoding representation.

[0161] 10.3.4 Linear classification layer maps Attn to the first prediction vector logits of the category number dimension, performs softmax operation on logits, and obtains T m Corresponding to the probability of each category in the category table input by the user, the category with the highest probability is taken as T m Category cat2.

[0162] 10.3.5 The linear classification layer uses the cross-loss entropy function to calculate T' and T # The distribution gap between them is used to obtain the loss value Loss. The Adam optimization algorithm (see the literature "A method for stochastic optimization", 2015, a stochastic optimization method) is used to minimize the Loss. The weight parameters of the feedforward neural network layer and the linear mapping layer are updated once through back propagation to obtain the updated W F and W Y .

[0163] 10.3.6 Let m = m + 1. If m ≤ M, go to 10.3.3. If m>M, it means that all M data samples of T have participated in the training and one iteration of training has been completed. Go to 10.3.7.

[0164] 10.3.7 Let epoch = epoch + 1. If epoch ≤ iteration threshold θ, go to 10.3.2; otherwise, it means that the training meets the iteration number requirement, the training is completed, and the weight parameters of the feedforward neural network layer and the linear mapping layer in the second-stage classification module are obtained, and go to step 12.

[0165] In the eleventh step, the training method described in the tenth step is adopted, and the second training set RR is used to perform supervised training on the two-stage classification system to obtain the network weight parameters of the feedforward neural network layer and the linear classification layer in the second-stage classification module, and then go to the twelfth step.

[0166] In the twelfth step, the weight parameters obtained through training are assigned to the second-stage classification module, that is, the trained W F As the network weight parameter of the feedforward neural network layer of the second-stage classification module, the trained W Y As the weight parameter of the linear classification layer of the second-stage classification module, a trained two-stage classification system is obtained.

[0167] Step 13: If you need to classify text data, go to step 14. If you need to classify source code data, go to step 15.

[0168] In the fourteenth step, the trained two-stage classification system classifies the text X input by the user to obtain the category to which the text X belongs. The method is:

[0169] 14.1 The preprocessing module receives the text X input by the user, X={x1,x2,···,x i ,···,x I}(where x i represents the i-th word in the input source code), use the word-level encoding method described in step 4.1.2 to encode x1, x2, ···, x in X. i ,···,x I Perform word mapping to obtain the word-level encoding sequence set G(X) of X, G(X) = {G(x1), G(x2), ··G(x i ),···,G(x I )},G(x i ) is x i The word-level encoding is performed and G(X) is sent to the first-stage classification module and the second-stage classification module.

[0170] 14.2 The first stage classification module receives G(X) from the preprocessing module, classifies G(X), and obtains the probability set P of each category in the category table input by the user corresponding to X, P = {p1, p2, ..., p u ,...,p U}(where p u It represents the probability that the first-stage classification model predicts that X belongs to the category corresponding to the subscript u, 1≤u≤U). The category corresponding to the highest probability is taken as the category cat1 of X, and the obtained probability set P and category cat1 are sent to the boundary sample screening module.

[0171] 14.3 The boundary sample screening module receives P and cat1 from the first classification module, receives the easily confused category binary sequence L from the easily confused category identification module, and determines whether X needs to be reclassified using the second stage classification module based on L, P and cat1. The method is:

[0172] 14.3.1 Let e ​​= 1, and take the tuple l from L e .

[0173] 14.3.2 If cat1 does not belong to the tuple l e In the category, go to 14.3.3; if category cat1 belongs to the tuple l e The categories in the table indicate that u corresponds to l e The subscript of the first category in , u′ corresponds to le For the subscript of the second category, go to 14.3.5.

[0174] 14.3.3 Let e ​​= e + 1. If e ≤ E, go to 14.3.2. If e > E, it means that cat1 has been compared with every pair in L, and cat1 does not belong to any pair of pairs in L. Go to 14.3.4.

[0175] 14.3.4 Take cat1 as the predicted category of X, output cat1, and go to step 16.

[0176] 14.3.5 Calculate the probability interpolation Δp = |p u -p u′ |.

[0177] 14.3.6 Let the reclassification threshold ω = 0.2. If Δp>ω, take cat1 as the category of X, output cat1, and go to step 16; if Δp<ω, set the signal flag that needs to reclassify G(X) to 1 and send it to the second stage classification module, and go to 14.4.

[0178] 14.4 The second stage classification module receives the signal flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is classified and the category corresponding to the highest probability is taken as the category cat2 of X. The method is:

[0179] 14.4.1 The feedforward neural network layer receives the word-level code G(X) from the preprocessing module, uses the nonlinear mapping method to perform nonlinear mapping on G(X), obtains the nonlinear mapped representation Attn′ of G(X), and sends Attn′ as the sentence-level code representation to the linear classification layer.

[0180] 14.4.2 The linear classification layer maps Attn′ to the second prediction vector logits′ in the dimension of the number of categories, performs a softmax operation on logits′, obtains the probability of each category in the category table corresponding to X, takes the category with the highest probability as the predicted label cat2 of X, and goes to step 16.

[0181] Step 15: The trained two-stage classification system classifies the source code X input by the user to obtain the category to which the source code X belongs. The method is:

[0182] 15.1 The preprocessing module receives the source code X input by the user, X = {x1, x2, ···, x i ,···,x I}(where x i represents the i-th word in the input source code), use the word-level encoding method described in step 4.1.2 to encode x1, x2, ···, x in X.i ,···,x I Perform word mapping to obtain the word-level encoding sequence set G(X) of X, G(X) = {G(x1), G(x2), ··G(x i ),···,G(x I )},G(x i ) is x i The word-level encoding is performed and G(X) is sent to the first-stage classification module and the second-stage classification module.

[0183] 15.2 The first stage classification module receives the word-level encoding G(X) of X from the preprocessing module, classifies G(X), and obtains the probability set P of each category in the category table corresponding to X, P = {p1, p2, ..., p u ,...,p U}(where p u It represents the probability that the first-stage classification model predicts that X belongs to the category corresponding to the subscript u, 1≤u≤U). The category corresponding to the highest probability is taken as the category cat1 of X, and the obtained probability set P and category cat1 are sent to the boundary sample screening module.

[0184] 15.3 The boundary sample screening module receives P and cat1 from the first classification module, and receives the second easily confused category binary sequence LL from the easily confused category identification module, and determines whether X needs to be reclassified using the second stage classification module through LL, P and cat1. The method is:

[0185] 15.3.1 Let e ​​= 1, and take the tuple ll from LL e .

[0186] 15.3.2 If cat1 does not belong to the tuple l e In the category, go to 15.3.3; if category cat1 belongs to the tuple l e The categories in the table indicate that u corresponds to l e The subscript of the first category in , u′ corresponds to l e For the subscript of the second category, go to 15.3.5.

[0187] 15.3.3 Let e=e+1. If e≤E, go to 15.3.2. If e>E, it means that cat1 has been compared with every pair in L, and cat1 does not belong to any pair of pairs in L. Go to 15.3.4.

[0188] 15.3.4 Take cat1 as the predicted category of X, output cat1, and go to step 16.

[0189] 15.3.5 Calculate the probability interpolation Δp = |p u -p u′|.

[0190] 15.3.6 Let the reclassification threshold ω = 0.2.2. If Δp>ω, take cat1 as the category of X, output cat1, and go to step 16; if Δp<ω, set the signal flag that needs to reclassify G(X) to 1 and send it to the second stage classification module, and go to 15.4.

[0191] 15.4 The second stage classification module receives the signal flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is classified to obtain the predicted class cat2 of X. The method is:

[0192] 15.4.1 The feedforward neural network layer receives the word-level code G(X) from the preprocessing module, uses the nonlinear mapping method to perform nonlinear mapping on G(X), obtains the nonlinear mapped representation Attn′ of G(X), and sends Attn′ as the sentence-level code representation to the linear classification layer.

[0193] 15.4.2 The linear classification layer maps Attn′ to the second prediction vector logits′ in the dimension of the number of categories, performs a softmax operation on logits′, obtains the probability of each category in the category table corresponding to X, takes the category with the highest probability as the predicted label cat2 of X, and goes to step 16.

[0194] Step 16. End.

[0195] In order to verify the effect of the present invention, the following experiment was conducted. The parameter settings during the experiment were consistent with those in the above-mentioned embodiment 1. The implementation process was:

[0196] The experiment was conducted on a computer with an 8-core Intel CPU and a 64GB physical memory and an NVIDIA V100 GPU. The operating system was Ubuntu 18.0.4 and the main coding language was Python. This experiment was tested on two examples, namely, a software engineering-related sentiment classification task (X is text) and a vulnerability classification task (X is source code). In the experiment, when X is a text, X is a text sequence of the form "Very very weird, maybe you can debug and provide source code of module?" and "This is a nice way to wrap up this logic, and good explanation in the comment above.", which respectively means "Very very weird, maybe you can debug and provide source code of module?" and "This is a nice way to wrap up this logic, and good explanation in the comment above." The first sentence expresses the emotion of angry, and the second sentence expresses the emotion of love. The present invention detects two pairs of easily confused categories (anger, fear) and (love, happiness) for the text X (belonging to the software engineering emotion classification task). When X is the first sentence, the SEntiMoji method of the background technology classifies it as the emotion of fear, and the result cat1 classified by the first stage classification module in step 14 of the present invention is also the emotion of fear, but because the second stage classification module in step 14.4 performs a second classification, cat2 is obtained as the emotion of anger. Similarly, when X is the second sentence, the SEntiMoji method of the background art classifies it as a happy emotion, and the result cat1 classified by the first-stage classification module in step 14 of the present invention is also a happy emotion, but because the second-stage classification module in step 14.4 performs a second classification, cat2 is obtained as a love emotion.When X is source code, X is a source code sequence in the form of "bool GesturePoint::IsInSecondClickTimeWindow()const{double duration=last_touch_time_-last_tap_time_;return duration<kMaximumSecondsBetweenDoubleClick;}". This code is the implementation of the member function IsInSecondClickTimeWindow() of a C++ class GesturePoint. The main purpose of this function is to determine whether the time interval between two click inputs is within an allowed time window, belonging to Input validation error (i.e., input validation error). Using the present invention to identify a pair of confusing categories for source code like X (belonging to vulnerability classification) is (input validation error, privilege escalation). At this time, the BERT method in the background technology classifies it as an error of Privilege escalation, and the result cat1 classified by the classification module in the first stage of the fifteenth step of the present invention is also an error of privilege escalation. However, due to the second classification by the classification module in the second stage of step 15.4, cat2 is obtained as an input validation error. The F1 value of the confusing categories using the present invention has a significant relative improvement and does not reduce the average F1 value of all categories. For the text classification task (taking the software engineering sentiment recognition task as an example), when the present invention classifies the angry category, the F1 value is 0.5652, while the F1 value of the SEntiMoji method is only 0.5379, and the present invention relatively increases by 5.08%; when the present invention classifies the love category, the F1 value is 0.6742, while the F1 value of the SEntiMoji method is only 0.6118, and the F1 value relatively increases by 10.2%; when the present invention classifies the happy category, the F1 value is 0.6182, while the F1 value of the SEntiMoji method is only 0.6036, and the F1 value relatively increases by 2.42%; when the present invention classifies the fear category, the F1 value is 0.5797, while the F1 value of the SEntiMoji method is only 0.4839, and the F1 value relatively increases by 19.8%; using the present invention makes the average F1 value of all categories in the software engineering sentiment recognition task category table relatively increase by 5.7%.For the source code classification task (taking the vulnerability classification task as an example), when the present invention classifies the input verification error class, the F1 value is 0.5188, while the F1 value of the BERT method is only 0.5188, and the F1 value is relatively improved by 8.62%; when the present invention classifies the privilege escalation class, the F1 value is 0.5707, while the F1 value of the BERT method is only 0.5707, and the F1 value is relatively improved by 2.46%; the use of the present invention makes the average F1 value of all categories in the vulnerability classification task category table relatively improved by 1.91%.

[0197] Table 1. Effect of the present invention on text classification tasks (taking software engineering sentiment recognition tasks as an example)

[0198]

[0199]

[0200] Table 2. Effect of the present invention on source code classification tasks (taking vulnerability classification tasks as an example)

[0201]

Claims

1. A method for classifying easily confused categories based on a two-stage model, characterized in that The following steps are involved: The first step is to build a two-stage classification system; the two-stage classification system consists of a preprocessing module, a first-stage classification module, an easily confused category recognition module, a boundary sample screening module, and a second-stage classification module; The preprocessing module is connected to the first stage classification module and the second stage classification module; during training, the first training set T or the second training set RR is read, T = {T 1 ,T 2 ,···T m ,···,T M }, where T m represents the mth text sequence in T, 1≤m≤M, M is the total number of texts in T, Indicates T m The i-th word of T m The total number of words in the 1 ,R 2 ,···R n ,···,R N }, where R n represents the nth source code sequence in RR, 1≤n≤N, N is the total number of source codes in RR; Represents R n The i-th word of the training set, 1≤i≤I; the preprocessing module uses the word embedding method to map each word in the training set to a word-level code, and obtains a word-level code sequence set of the training set. The preprocessing module sends the word-level code sequence set of the training set to the second-stage classification module; When classifying the text or source code input by the user, the preprocessing module receives the text or source code sequence X input by the user, X = {x1, x2, ..., x j ,...,x J }, where x j represents the jth word in X, 1≤j≤J, J is the number of words in X, and each word is mapped to a word-level code using the word embedding method to obtain the user word-level code sequence set G(X), G(X) = {G(x1), G(x2), ..., G(x j ),...,G(x J )}, where G(x j ) represents x j After the word level encoding mapped by the word embedding method, the preprocessing module sends G(X) to the first stage classification module and the second stage classification module; When it is necessary to obtain the easily confused category binary, the preprocessing module reads the test set, maps each word in the test set to a word-level code using the word embedding method, obtains the word-level code sequence set of the test set, and sends the word-level code sequence set of the test set to the first-stage classification module; The first-stage classification module is connected to the preprocessing module, the easily confused category identification module, and the boundary sample screening module, and receives the category table input by the user. The category table consists of two fields: the category name and the category subscript. The first-stage classification module receives the test set word-level coding sequence set and G(X) from the preprocessing module, classifies the test set word-level coding sequence set, obtains the confusion matrix, and sends the confusion matrix and the category table to the easily confused category identification module. The first-stage classification module classifies G(X) to obtain the probability set P of each category in the category table input by the user corresponding to X, P = {p1, p2, ..., p u ,...,p U }, where p u It means that the first stage classification module predicts the probability that G(X) belongs to the category corresponding to the subscript u, takes the category corresponding to the highest probability as the category cat1 of X, and sends P and cat1 to the boundary sample screening module; The easily confused category identification module is connected to the first-stage classification module and the boundary sample screening module, receives the category table and confusion matrix from the first-stage classification module, arranges all the non-diagonal values ​​in the confusion matrix from small to large, obtains the third quartile Q3, calculates the threshold for judging the easily confused category through Q3, compares all the non-diagonal elements in the confusion matrix with the easily confused category threshold, obtains the easily confused category binary sequence, and sends the easily confused category binary sequence and the category table to the boundary sample screening module; The boundary sample screening module is connected to the first-stage classification module, the easily confused category recognition module, and the second-stage classification module. It receives P and cat1 from the first-stage classification module, receives the easily confused category binary sequence and category table from the easily confused category recognition module, sets the reclassification threshold as ω, compares category cat1 with the easily confused category binary sequence, and if category cat1 does not belong to a category in a binary in the easily confused category binary sequence, then cat1 is taken as the category of X; if category cat1 belongs to a category in a binary in the easily confused category binary sequence, then the probability interpolation Δp is calculated; if Δp>ω, then category cat1 is taken as the category of X; If Δp<ω, the signal flag indicating that X needs to be reclassified is set to 1, and the flag is sent to the second-stage classification module; The second-stage classification module is composed of a feedforward neural network layer and a linear classification layer; the feedforward neural network layer is connected to the preprocessing module, the linear classification layer, and the boundary sample screening module; during training, the feedforward neural network layer receives the training set word-level coding sequence set from the preprocessing module, performs nonlinear mapping on each word-level coding in the training set word-level coding sequence set, obtains the representation Attn after the nonlinear mapping of the training set word-level coding sequence set, and sends Attn to the linear classification layer; when classifying the text or source code input by the user, the feedforward neural network layer receives the flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, nonlinear mapping is performed on G(X) to obtain the representation Attn' after the nonlinear mapping of G(X), and Attn' is sent to the linear classification layer; The linear classification layer is connected to the feedforward neural network layer. During training, the linear classification layer receives Attn from the feedforward neural network layer, performs linear mapping on Attn, obtains the first prediction vector, i.e., logits, performs Softmax operation on logits, and obtains the probability set P' for each category in the category table corresponding to each sample in the training set, P'={p'1,p'2,...,p' u ,...,p' U }, where p' u Represents the second stage classification model prediction T m The probability of belonging to the category corresponding to the subscript u, taking the category corresponding to the highest probability as the category cat2 of each sample in the training set, and using the cross loss entropy function to calculate the loss Loss to perform back propagation to update the model parameters; When classifying the text or source code input by the user, the linear classification layer receives Attn' from the feedforward neural network layer, performs linear mapping on Attn', obtains the second prediction vector logits', performs Softmax operation on logits', and obtains the second probability set PP' for each category in the category table corresponding to G(X), PP'={pp'1,pp'2,...,pp' u ,...,pp' U }, where pp' u It indicates the probability that the second-stage classification model predicts that X belongs to the category corresponding to the subscript u. The category corresponding to the highest probability is taken as the category cat2' of X, and the category cat2' is output; The second step is to prepare the dataset for training the two-stage classification system by: 2.1 Use the text data in the Polarity dataset to construct the first training set and the first test set for the software engineering sentiment classification task; put the AA samples in the Polarity dataset into the first training set, and the BB samples into the first test set; each sample in the Polarity dataset contains a text and its corresponding actual label; the text set in the first training set constitutes the first training set set T, and the actual label sequence set in the first training set constitutes the first training set actual label sequence set T # ; Let T = {T 1 ,T 2 ,···T m ,···,T M }, where T m represents the mth text sequence in T, where 1≤m≤M, and M is the total number of texts in T; Indicates T m The i-th word of the first test set, 1≤i≤I; the text set in the first test set constitutes the first test set V, V={V 1 ,V 2 ,···V r ,···,V R }, where V r represents the rth text in V, where 1≤r≤R, R is the total number of texts in V, in Indicates V r The qth word in V, 1≤q≤Q, Q is r The actual label sequence set in the first test set samples constitutes the actual label sequence set V of the first test set # ; AA and BB are positive integers; 2.2 Use the source code data in the MSR dataset to construct the second training set and the second test set for the software engineering sentiment classification task; put the CC samples in the MSR dataset into the training set, and the DD samples into the second test set; each sample in the MSR dataset contains a source code and its corresponding actual label; the source code set in the second training set constitutes the second training set set RR, and the actual label sequence set in the second training set constitutes the second training set actual label sequence set RR # ; Let RR = {R 1 ,R 2 ,···R n ,···,R N }, where R n represents the nth source code sequence in RR, 1≤n≤N, N is the total number of source codes in RR; Represents R n The source code set in the second test set constitutes the second test set S, S = {S 1 ,S 2 ,···S o ,···,S O }, where S o represents the o-th text in S, where 1≤o≤O, O is the total number of source codes in S, in Indicates S o The qqth word in S, 1≤qq≤QQ, QQ is S o The actual label sequence set in the second test set samples constitutes the actual label sequence set S of the second test set # ; CC and DD are positive integers; Step 3: If you need to classify the text data, go to step 4; If you need to classify the source code data, go to step 5; In the fourth step, the preprocessing module uses a preprocessing method to map the text data in T to the first training set word-level coding sequence set G(T), and maps the text data in V to the first test set word-level coding sequence set G(V), the method is: 4.1 Use the word-level mapping method to map the M text data in T to G(T), the method is: 4.1.1 Let variable m = 1; 4.1.2 Read the mth text T in T m , using word-level encoding method to m of Perform word mapping and get T m The word-level encoding sequence G(T m ), the method is: 4.1.2.1 Let variable i = 1; 4.1.2.2 T m The i-th word in is mapped to the word-level encoding 4.1.2.3 Let i=i+1. If i≤I, go to 4.1.2.

2. If i>I, then T m The I words in T have been mapped to word-level codes, and the word-level code sequence of the mth text data in T is obtained. Indicates T m For the i-th word, go to 4.1.3; 4.1.3 Let m = m + 1. If m ≤ M, go to 4.1.

2. If m > M, it means that the M text data of T have been mapped to word-level coding sequences, and G(T) is obtained. G(T) = {G(T 1 ),G(T 2 ),...,G(T m ),...,G(T M )},G(T m ) represents the word-level encoding sequence of the mth text data in T; 4.2 Use the word-level mapping method described in 4.1 to map the R text data in V into the first test set word-level encoding set G(V); 4.3 Send G(T) to the second-stage classification module, send G(V) to the first-stage classification module, and go to step 6; Step 5: The preprocessing module uses the preprocessing method described in step 4 to map the source code data in RR to the second training set word-level coding sequence set G(RR), and maps the source code data in S to the second test set word-level coding sequence set G(S), sends G(RR) to the second-stage classification module, sends G(S) to the first-stage classification module, and then goes to step 7; In the sixth step, the first stage classification module receives the category table input by the user, receives G(V) from the preprocessing module, classifies G(V) using a classification method, and obtains the first confusion matrix CM, which is: 6.1 The first stage classification module receives the category table input by the user and receives G(V) from the preprocessing module; the category table consists of two fields: category name and category subscript, and U is the total number of category names in the category table; 6.2 The first stage classification module classifies G(V) and obtains the category set pred; 6.3 The first stage classification module uses the confusion_matrix function to obtain the first confusion matrix CM, CM = {p 11 ,p 12 ,...p uu' ,...,p UU ,}, where p uu' It indicates the probability that the first-stage classification module predicts the category corresponding to the subscript u as the category corresponding to the subscript u', 1≤u≤U, 1≤u'≤U; 6.4 The first stage classification module sends the CM and category table to the easily confused category identification module, and then goes to step 8; Step 7: The first-stage classification module receives the category table input by the user, receives G(S) from the preprocessing module, classifies G(S) using the classification method described in Step 6, obtains the second confusion matrix CM', sends CM' and the category table to the easily confused category recognition module, and then goes to Step 9; In the eighth step, the easily confused category identification module adopts an identification method and uses the first confusion matrix CM to obtain the first easily confused category binary sequence L, the method is: 8.1 Initialize the first easily confused category binary sequence L to be empty; 8.2 Arrange all non-diagonal values ​​in the first confusion matrix CM from small to large; 8.3 Calculate the third quartile Q3 of the arranged CM; 8.4 Let the easily confused category threshold δ = 2Q3; 8.5 Let variable u = 1; 8.6 Let variable u' = 1; 8.7 Comparison of p uu' and δ; if p uu' >δ, then the category corresponding to the subscript u and the category corresponding to the subscript u' are a set of easily confused category tuples, and the tuple (category corresponding to u, category corresponding to u') is added to L; 8.8 Let u'=u'+1. If u'≤U, go to 8.

7. If u'>U, it means that a row in CM has been traversed, go to 8.

9. 8.9 Let u=u+1. If u≤U, go to 8.

7. If u>U, it means that CM has been completely traversed and the first confusing category binary sequence L is obtained, L={l1,l2,...,l e ,...,l E }, where l e is a pair of easily confused category binary tuples, 1≤e≤E, E is the number of binary tuples identified by the easily confused category identification module, L is sent to the boundary sample screening module, and go to step 10; Step 9: The easily confused category recognition module adopts the recognition method described in step 8, uses the second confusion matrix CM' to obtain the second easily confused category binary sequence LL, sends LL to the boundary sample screening module, and then goes to step 11; The tenth step is to use the training method to train the two-stage classification system in a supervised manner using T to obtain the network weight parameters of the feedforward neural network layer and the linear classification layer of the second-stage classification module. The method is: 10.1 Initialize the weight parameters and set the parameter set W of the feedforward neural network layer F Initialize the elements in to random numbers in [0,1]; set the parameter set W of the linear classification layer Y The elements in are initialized to random numbers in [0,1]; 10.2 Set network training hyperparameters, including word embedding size, network model learning rate, weight decay rate weight_decay, number of heads, warmup learning rate warmup, batch size batch_size, and encoding dropout rate; 10.3 The second stage classification module receives G(T) from the preprocessing module and uses supervised iterative training. Each round of training obtains the first predicted label sequence T' and calculates T' and T # The distribution gap is obtained, the loss value is minimized and the network parameters of the feedforward neural network layer and the linear classification layer are updated until the number of iterations is met, and the weight parameters of the feedforward neural network layer and the linear classification layer are obtained. The method is as follows: 10.3.1 Initialize the training iteration parameter epoch = 1; initialize the iteration threshold θ to a positive integer; 10.3.2 Let variable m = 1; 10.3.3 Feedforward neural network layer uses nonlinear mapping method to G(T m ) is nonlinearly mapped to obtain G(T m ) The representation Attn after nonlinear mapping is sent to the linear classification layer as the sentence-level encoding representation; 10.3.4 Linear classification layer maps Attn to the first prediction vector logits of the category number dimension, performs softmax operation on logits, and obtains T m Corresponding to the probability of each category in the category table input by the user, the category with the highest probability is taken as T m Category cat2; 10.3.5 The linear classification layer uses the cross-loss entropy function to calculate T' and T # The distribution gap between them is used to obtain the loss value Loss, and the Adam optimization algorithm is used to minimize the Loss. The weight parameters of the feedforward neural network layer and the linear mapping layer are updated through back propagation to obtain the updated W F and W Y ; 10.3.6 Let m = m + 1. If m ≤ M, go to 10.3.

3. If m > M, it means that all M data samples of T have participated in the training and one iteration of training has been completed. Go to 10.3.

7. 10.3.7 Let epoch = epoch + 1. If epoch ≤ iteration threshold θ, go to 10.3.

2. Otherwise, it means that the training meets the iteration number requirement, and the training is finished. The weight parameters of the feedforward neural network layer and the linear mapping layer in the second-stage classification module are obtained, and go to step 12. Step 11: adopt the training method described in step 10, use the second training set RR to train the two-stage classification system in a supervised manner, obtain the network weight parameters of the feedforward neural network layer and the linear classification layer in the second-stage classification module, and then go to step 12; In the twelfth step, the weight parameters obtained through training are assigned to the second-stage classification module, that is, the trained W F As the network weight parameter of the feedforward neural network layer of the second-stage classification module, the trained W Y As the weight parameter of the linear classification layer of the second-stage classification module, a trained two-stage classification system is obtained; Step 13: If you need to classify text data, go to step 14; if you need to classify source code data, go to step 15; In the fourteenth step, the trained two-stage classification system classifies the text X input by the user to obtain the category to which the text X belongs. The method is: 14.1 The preprocessing module receives the text X input by the user, X={x1,x2,···,x i ,···,x I }, at this time x i represents the i-th word in the input source code, and uses the word-level encoding method described in step 4.1.2 to encode x1, x2, ···, x in X i ,···,x I Perform word mapping to obtain the word-level encoding sequence set G(X) of X, G(X) = {G(x1), G(x2), ··G(x i ),···,G(x I )},G(x i ) is x i The word-level encoding is performed and G(X) is sent to the first-stage classification module and the second-stage classification module; 14.2 The first stage classification module receives G(X) from the preprocessing module, classifies G(X), and obtains the probability set P of each category in the category table input by the user corresponding to X, P = {p1, p2, ..., p u ,...,p U }, where p u It indicates the probability that the first-stage classification model predicts that X belongs to the category corresponding to the subscript u. The category corresponding to the highest probability is taken as the category cat1 of X. The obtained probability set P and category cat1 are sent to the boundary sample screening module. 14.3 The boundary sample screening module receives P and cat1 from the first classification module, receives the easily confused category binary sequence L from the easily confused category identification module, and determines whether X needs to be reclassified using the second stage classification module based on L, P and cat1; the method is: 14.3.1 Let e ​​= 1, and take the tuple l from L e ; 14.3.2 If cat1 does not belong to the tuple l e In the category, go to 14.3.3; if category cat1 belongs to the tuple l e The categories in the table indicate that u corresponds to l e The subscript of the first category in the , u' corresponds to l e For the subscript of the second category in the , go to 14.3.5; 14.3.3 Let e ​​= e + 1. If e ≤ E, go to 14.3.

2. If e > E, it means that cat1 has been compared with every pair in L, and cat1 does not belong to any pair of pairs in L. Go to 14.3.

4. 14.3.4 Take cat1 as the predicted category of X, output cat1, and go to step 16; 14.3.5 Calculate the probability interpolation Δp = |p u -p u' |; 14.3.6 Let the reclassification threshold ω be a decimal between 0 and 1; if Δp>ω, take cat1 as the category of X, output cat1, and go to step 16; if Δp<ω, set the signal flag that needs to reclassify G(X) to 1 and send it to the second stage classification module, and go to 14.4; 14.4 The second stage classification module receives the signal flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is classified and the category corresponding to the highest probability is taken as the category cat2 of X. The method is: 14.4.1 The feedforward neural network layer receives the word-level encoding sequence G(X) from the preprocessing module, uses the nonlinear mapping method to perform nonlinear mapping on G(X), obtains the nonlinearly mapped representation Attn' of G(X), and sends Attn' as the sentence-level encoding representation to the linear classification layer; 14.4.2 The linear classification layer maps Attn' to the second prediction vector logits' of the category number dimension, performs a softmax operation on logits', obtains the probability of each category in the category table corresponding to X, takes the category with the highest probability as the predicted label cat2 of X, and goes to step 16; Step 15: The trained two-stage classification system classifies the source code X input by the user to obtain the category to which the source code X belongs. The method is: 15.1 The preprocessing module receives the source code X input by the user, X = {x1, x2, ···, x i ,···,x I }, at this time x i represents the i-th word in the input source code, and uses the word-level encoding method described in step 4.1.2 to encode x1, x2, ···, x in X i ,···,x I Perform word mapping to obtain the word encoding sequence set G(X) of X, G(X) = {G(x1), G(x2), ··G(x i ),···,G(x I )},G(x i ) is x i The word-level encoding is performed and G(X) is sent to the first-stage classification module and the second-stage classification module; 15.2 The first stage classification module receives the word-level encoding G(X) of X from the preprocessing module, classifies G(X), and obtains the probability set P of each category in the category table corresponding to X, P = {p1, p2, ..., p u ,...,p U }, p u It indicates the probability that the first-stage classification model predicts that X belongs to the category corresponding to the subscript u. The category corresponding to the highest probability is taken as the category cat1 of X. The obtained probability set P and category cat1 are sent to the boundary sample screening module. 15.3 The boundary sample screening module receives P and cat1 from the first classification module, receives the second easily confused category binary sequence LL from the easily confused category identification module, and determines whether X needs to be reclassified using the second stage classification module based on LL, P and cat1; the method is: 15.3.1 Let e ​​= 1, and take the tuple ll from LL e ; 15.3.2 If cat1 does not belong to the tuple l e In the category, go to 15.3.3; if category cat1 belongs to the tuple l e The categories in the table indicate that u corresponds to l e The subscript of the first category in , u' corresponds to l e For the subscript of the second category, go to 15.3.5; 15.3.3 Let e ​​= e + 1. If e ≤ E, go to 15.3.

2. If e > E, cat1 has been compared with every pair in L, and cat1 does not belong to any pair in L. Go to 15.3.

4. 15.3.4 Take cat1 as the predicted category of X, output cat1, and go to step 16; 15.3.5 Calculate the probability interpolation Δp = |p u -p u' |; 15.3.6 Let the reclassification threshold ω be a decimal between 0 and 1; if Δp>ω, take cat1 as the category of X, output cat1, and go to step 16; if Δp<ω, set the signal flag that needs to reclassify G(X) to 1 and send it to the second stage classification module, and go to 15.4; 15.4 The second stage classification module receives the signal flag from the boundary sample screening module and receives G(X) from the preprocessing module. If the signal flag value is 1, G(X) is classified to obtain the predicted class cat2 of X. The method is: 15.4.1 The feedforward neural network layer receives the word-level code G(X) from the preprocessing module, uses the nonlinear mapping method to perform nonlinear mapping on G(X), obtains the representation Attn' of G(X) after nonlinear mapping, and sends Attn' as the sentence-level code representation to the linear classification layer; 15.4.2 The linear classification layer maps Attn' to the second prediction vector logits' of the category number dimension, performs a softmax operation on logits', obtains the probability of each category in the category table corresponding to X, takes the category with the highest probability as the predicted label cat2 of X, and goes to step 16; Step 16. End.

2. The method for classifying easily confused categories based on a two-stage model as claimed in claim 1, characterized in that If X received by the preprocessing module from the user is text, the first-stage classification module is any one of ML, RNN, LSTM, and BERT; if X received by the preprocessing module from the user is source code, the first-stage classification module is any one of ML, GGNN, LSTM, and CodeBERT.

3. The method for classifying easily confused categories based on a two-stage model as described in claim 1 or 2, characterized in that If the object X to be classified is text, when the first-stage classification module is any one of ML, RNN, and LSTM, the second-stage classification module is BERT; when the first-stage classification module is BERT, the second-stage classification module is RoBERTa; if the object X to be classified is source code, when the first-stage classification module is any one of ML, GraphNet, and LSTM, the second-stage classification module is CodeBERT; when the first-stage classification module is CodeBERT, the second-stage classification module is GraphCodeBERT.

4. The method for classifying easily confused categories based on a two-stage model as claimed in claim 1, characterized in that In the second step, AA is 15436, BB is 349, CC is 6670, and DD is 845.

5. The method for classifying easily confused categories based on a two-stage model as claimed in claim 1, characterized in that The method for setting the network training hyperparameters in step 10.2 is: set the word embedding size to 768 and the network model learning rate to 5×10 -5 , set the weight decay rate weight_decay value to 0.01, the number of heads head to 12, the warmup learning rate warmup to 0.05, the batch size batch_size to 16, and the encoding dropout rate dr opout to 0.

5.

6. The method for classifying easily confused categories based on a two-stage model as claimed in claim 1, characterized in that The iteration threshold θ described in step 10.3.1 is initialized to 20.

7. The method for classifying easily confused categories based on a two-stage model as claimed in claim 1, characterized in that The reclassification threshold ω described in steps 14.3.6 and 15.3.6 is 0.2.

Images