An automatic classification method for software vulnerabilities based on weighted word vectors and neural networks

Abstract

The invention discloses a method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks, which belongs to the field of computer software security and includes the following steps: S1, data preprocessing of a text describing a detailed description of a vulnerability; S2, after the vulnerability text is preprocessed, weighted word vectors are constructed by combining a Word2Vec model and an N‑TF‑IDF algorithm to complete text vector representation; S3, automatic classification of vulnerabilities is realized by a TCNN‑BiGRU vulnerability classification model based on a neural network, and the model mainly includes four parts, namely, an input layer, a TextCNN structure, a BiGRU structure, and a fully connected layer. The invention can meet the demand for automatic classification of software vulnerabilities, and is not limited by the increase in the number of vulnerabilities and the types of vulnerabilities, effectively handles the high dimensionality and sparsity problems of word vector space, and shows good performance in multidimensional evaluation indicators such as accuracy, macro precision, macro recall, and macro F1 value.

Description

Technical Field

[0001] The present invention belongs to the field of computer software security and relates to a method for automatically classifying software vulnerabilities based on weighted word vectors and neural networks. Background Art

[0002] In recent years, the number and types of computer software vulnerabilities have gradually increased. The National Vulnerability Database (NVD) of the United States reported a total of 18,378 vulnerabilities in 2021, setting a record high. The effective analysis and management of vulnerabilities has become increasingly important. Vulnerability classification is an important activity in software development and software quality maintenance. Different types of vulnerabilities have different degrees of harm. By extracting characteristic attributes from vulnerability descriptions and classifying vulnerabilities in a timely manner, the cost of vulnerability management and system maintenance can be effectively reduced. Research on automated vulnerability classification models has emerged as the times require.

[0003] There are many early computer security vulnerability classification methods, including RISOS classification, PA classification, Aslam classification, etc. The RISOS classification method describes the attack method of security vulnerabilities, mainly studying operating system vulnerabilities, and it is difficult to deal with various existing complex vulnerabilities; the PA classification method further discusses the security vulnerabilities of applications on the operating system based on RISOS; the Aslam classification method corresponds to the UNIX operating system, and considers system kernel-level vulnerabilities. Due to the endless emergence of computer security vulnerabilities in recent years, the location of vulnerabilities is no longer limited to operating systems and servers, but is mostly concentrated in Web browsers, data or system backup and recovery software, antivirus software, etc. Therefore, the early classification methods can no longer meet the requirements for classifying various existing vulnerabilities, and the traditional manual vulnerability classification method is no longer applicable. Vulnerability mining technology is more automated and intelligent.

[0004] Existing vulnerability automatic classification methods mostly use classic classification algorithms and improved algorithms of machine learning, which improve the accuracy of vulnerability automatic classification to a certain extent. Davari et al. proposed a vulnerability automatic classification framework based on activation vulnerability conditions, using common machine learning techniques to build a classifier with the highest F1-score to mark unknown vulnerabilities. This method performs well on a dataset of 580 software security vulnerabilities in the Firefox project. In order to improve the performance of the vulnerability automatic classification method based on support vector machines, Yang et al. used fuzzy entropy for feature selection, determined feature weights by TF-IDF, and optimized the parameters of the support vector machine using an improved particle swarm algorithm. Compared with the traditional support vector machine method, this method improves classification accuracy and shortens classification time. Jinfu Chen et al. proposed a vulnerability severity classification framework based on frequency-inverse gravity moment (TF-IGM) and feature selection. The superiority of the TF-IGM model and feature selection was demonstrated through experiments, and there were significant differences between different data sets.

[0005] However, as the number and complexity of vulnerability data increase, vulnerability features show the characteristics of high dimensionality and sparsity. Traditional vector representation and machine learning methods have shown limitations. Deep learning neural networks have gradually been applied to the task of automatic vulnerability classification. Han et al. proposed a deep learning-based method to predict the severity of software vulnerabilities. This method uses a word embedding model and a layer of shallow convolutional neural network to automatically capture the discriminant words and sentence features of vulnerability descriptions, and obtains high classification accuracy and training efficiency. Huang et al. proposed a vulnerability automatic classification method that combines a word vector space model with a deep neural network model. On the NVD dataset, it was proved that the classification performance of the deep neural network is better than that of machine learning classifiers such as SVM and KNN. However, the word vector space constructed by this method combined with TF-IDF and information gain is still sparse. Nakagawa et al. used the word2vec model for word embedding and used a character-level convolutional neural network (character-level CNN) to predict the severity of the vulnerability with an accuracy of 72.5%. Aghaei et al. proposed an automated technology using text mining technology and neural networks, using N-gram for feature selection and a layer of shallow convolutional neural network to achieve classification, with an accuracy of 92% on the NVD dataset and 75% and 90% for fine-grained and coarse-grained classification on the MITRE dataset, respectively. Since there is only one layer of shallow convolutional neural network, the performance of the algorithm is not compared with more complex networks. Das et al. proposed a Transformer-based framework (V2W-BERT), which combines the ideas of natural language processing, link prediction, and transfer learning, and performs well in both large and small samples. Vishnu et al. proposed a self-attention-based neural network (SA-DNN) model, which combines text mining methods to identify vulnerability categories from a large number of vulnerability description texts. The model was evaluated using 134,091 vulnerability reports to prove the effectiveness of the method. The above-mentioned neural network-based classification methods show better performance in dealing with the high dimensionality and sparsity of word vector space, but these methods all use a single neural network model, without considering the effectiveness of word vector weighted representation and the generalization ability of the model. The performance of automatic vulnerability classification needs to be improved. Summary of the invention

[0006] The purpose of the present invention is to provide a method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks. By improving the TF-IDF algorithm to construct a more effective weighted word vector, a word embedding model Word2Vec is used to generate a low-dimensional dense word embedding vector, and a convolutional neural network is combined with a recurrent neural network to construct a vulnerability automatic classification model TCNN-BiGRU, so as to achieve feature dimension reduction and effective classification of vulnerability text. At the same time, the Dropout method and the Early Stopping method are introduced to suppress the overfitting problem and improve the generalization ability of the model. The present invention uses the vulnerability detailed description information as a text data set and the corresponding vulnerability category as a text label, and mainly includes three parts: data preprocessing, text representation and TCNN-BiGRU classification model.

[0007] The present invention studies the automatic classification method of software vulnerabilities and conducts experimental analysis on the internationally recognized U.S. National Computer General Vulnerability Database NVD. The present invention can adapt to real-time updated software vulnerability data and effectively improve the performance of software vulnerability classification, which is particularly important for the effective classification management of the increasing number of software vulnerability types and quantities.

[0008] The present application discloses a method for automatically classifying software vulnerabilities based on weighted word vectors and neural networks, comprising the following steps:

[0009] S1, data preprocessing of vulnerability detailed description text, specifically including the following steps:

[0010] S11, word segmentation and punctuation removal, text segmentation can be achieved by identifying spaces and punctuation in the vulnerability description text;

[0011] S12, converting uppercase letters to lowercase and restoring morphology, in this step, different forms of English words are converted into a unified form to facilitate subsequent text processing;

[0012] S13, remove stop words and special symbols. Stop words include prepositions, articles, pronouns, etc. Special symbols are mainly characters in the text that have no semantic connection with the context;

[0013] S2, after the vulnerability text is preprocessed, the weighted word vector is constructed by combining the Word2Vec model and the N-TF-IDF algorithm to complete the text vector representation, which specifically includes the following steps:

[0014] S21, train the word vectors of all words in the text set through the Word2Vec model, which maps each word into a dense vector in a low-dimensional space, representing the relevant information between words at the semantic level. The present invention adopts the Skip-gram model in Word2Vec, which predicts the context based on the central word. The input is the one-hot vector of the central word, and the output is the prediction of the context of the central word;

[0015] S22, for the training set, generate a training set vocabulary, which includes all non-repeated words in the training set text, calculate the N-TF-IDF weights of all words in the vocabulary according to formulas (1)-(3), and then form weighted word vectors of all words in the training set vocabulary according to formula (4) combined with Word2Vec word vectors, so as to represent all texts in the training set as a sentence matrix composed of weighted word vectors;

[0016] The improved word frequency TF is the ratio of the word frequency within the category to the word frequency of all words in the category, as shown in formula (1):

[0017]

[0018] Among them, n i,j For word t i The number of occurrences in the jth class, ∑ m n m,j is the number of occurrences of all words in the jth category.

[0019] The improved IDF calculation formula introduces the number of category texts, as shown in formula (2):

[0020]

[0021] Where N is the total number of texts in the text set, N i,j is the category j that contains word t i The number of texts, ∑ k N i,k For all categories containing word t i The number of texts.

[0022] Combining formula (1) and formula (2), the importance of words in different categories of text is shown in formula (3):

[0023]

[0024] S23, for the validation set and the test set, generate a validation set vocabulary and a test set vocabulary respectively. If the word is in the training set vocabulary, its N-TF-IDF weight is set to the N-TF-IDF weight of the word in the training set. If the word is not in the training set vocabulary, its N-TF-IDF weight is set to 0.5. Then, the weighted word vectors of all words in the vocabulary are formed by combining the Word2Vec word vectors, so that all the texts in the validation set and the test set are represented as sentence matrices composed of weighted word vectors. Construct the weighted Word2Vec word vectors as follows:

[0025] The matrix M∈R is obtained by the N-TF-IDF algorithm |V|×|K| , where |V| is the number of all non-repeated words in the data set, |K| is the number of categories, that is, the matrix M is the weight of each word in different categories. The maximum value of the weight of each word in all categories is selected as the final weight of the word, and multiplied by its Word2Vec word vector to obtain the final weighted word vector of the word, as shown in formula (4):

[0026] Weight_Vector(t i )=Word2Vec(t i )*max(N - TF-IDF i,j ) (4)

[0027] Among them, Weight_Vector(t i ) is the word t i The weighted word vector weight, Word2Vec(t i ) is the word t i The Word2Vec word embedding vector, max() function is used to obtain the word t i The maximum value of weights among all categories.

[0028] S3, the TCNN-BiGRU vulnerability classification model based on neural network realizes automatic vulnerability classification. The model mainly consists of four parts, namely input layer, TextCNN structure, BiGRU structure and fully connected layer.

[0029] S31, the input layer input is a sentence matrix represented by word embedding vectors. Since the input of the convolutional layer requires the same length of sentences, it is necessary to do fixed-length processing, set the length to n, truncate the excess, and fill in 0 if it is less than the length. The matrix of the input layer is recorded as W n×d , where n is the number of words in the sentence and d is the dimension of the word vector.

[0030] S32, TextCNN structure consists of convolutional layers and pooling layers. In the convolutional layer, convolution kernels of different sizes are applied to extract features from the input matrix, taking into account not only the meaning of the word but also the word order and its context. The convolution kernel can be defined as ω∈R h×d , where h is the height of the convolution kernel, which is a hyperparameter, usually set to 2 to 5, and d is the width of the convolution kernel, which is the same as the dimension of the word vector. The convolution kernel performs a convolution operation on each possible window in the sentence vector matrix to generate a feature map S j ∈R n-h+1 , where n is the number of words in the sentence, and convolution kernels of different sizes will generate feature maps of different sizes.

[0031] Applying convolution kernels of different sizes in the convolution layer generates feature maps of different sizes, so it is necessary to perform a pooling operation on each feature map to obtain features of the same dimension. The most commonly used pooling functions are maximum pooling and average pooling. This paper uses maximum pooling to perform a pooling operation on the output of the convolution layer, extracting the maximum value of each feature map to represent the feature, thereby capturing the most important features. Through the maximum pooling operation, each convolution kernel obtains a feature value, and the pooling operation is performed on all convolution kernels, and then they are concatenated to obtain the final feature vector of the pooling layer.

[0032] S33 uses a bidirectional gated recurrent unit (BiGRU). BiGRU applies GRU twice in opposite directions, that is, at each time step t, the input data is passed to the forward hidden layer and the backward hidden layer at the same time. Finally, the output of BiGRU is obtained by concatenating the feature vectors output by the two unidirectional GRUs.

[0033] The core structure of GRU includes two gate structures: reset gate and update gate. The specific calculation formula of GRU internal structure is as follows:

[0034] z t =σ(W z ·[h t-1 ,x t ]) (5)

[0035] r t =σ(W r ·[h t-1 ,x t ]) (6)

[0036]

[0037]

[0038]

[0039]

[0040]

[0041] Among them, r t Represents the gate value of the reset gate, z t represents the gate value of the update gate, h t-1 is the output of the hidden layer at time t-1, x t is the input at time t, h t is the output of the hidden layer at time t, Represents the candidate value of the hidden layer node at time t, three W z , W r , W represents the linear transformation operation, * represents the matrix element multiplication, σ represents the sigmoid activation function, y t Indicates output.

[0042] and are the states of the forward and backward hidden layers at time step t, respectively. and are the states of the forward and backward hidden layers at time step t-1, h t is the final output of the BiGRU layer.

[0043] The feature vector output by the BiGRU layer is input into the pooling layer, and the maximum pooling operation is used to continue to extract the feature vector with the strongest representative ability, thereby reducing the dimension of the output vector.

[0044] S34, in order to suppress the overfitting problem of the model, the present invention adds a Dropout layer between the pooling layer and the fully connected layer of the TCNN-BiGRU model, and adds an Early Stopping method during the training process of the model.

[0045] S35, the input of the fully connected layer is the final vector obtained by concatenating the feature vectors output by the TextCNN structure and the BiGRU structure. The fully connected layer is similar to the hidden layer in a traditional neural network. Each neuron in it is fully connected to all neurons in the previous layer, integrating local feature maps to obtain global feature information, and sending the output value to the Softmax classifier. The Softmax activation function operates on the vector, maps the values ​​in the vector to values ​​between (0, 1), and obtains the probability that the sentence belongs to each category. The category with the largest probability value is the predicted classification result. The calculation formula of the Softmax function is as follows:

[0046]

[0047] Among them, p(y i|s) is the probability that the sentence belongs to the i-th category, y i is the i-th value of the vector, and c is the number of categories.

[0048] Due to the adoption of the above technical solution, the technical effects achieved by the present invention are as follows:

[0049] The present invention can meet the demand for automatic classification of software vulnerabilities and is not limited by the increase in the number and types of vulnerabilities.

[0050] The traditional TF-IDF algorithm assigns a weight to each word, but usually the importance of words in different categories of text is different. This paper proposes a new word weighting method N-TF-IDF, which can assign different weights to words based on the distribution between categories.

[0051] The present invention combines the N-TF-IDF algorithm with the Wrod2Vec word vector to construct a low-dimensional dense weighted word vector.

[0052] The present invention combines the convolutional neural network with the recurrent neural network in deep learning, fully utilizes the advantage of automatic feature extraction of neural networks, and combines the characteristics of TextCNN extracting local semantic information of vulnerability text with GRU capturing the long-term dependencies of vulnerability text context, and proposes a TCNN-BiGRU vulnerability classification model based on neural network.

[0053] The present invention adopts a bidirectional GRU algorithm. Compared with a unidirectional GRU algorithm that can only capture the features of the word before the current word, the bidirectional GRU algorithm can capture the features of the previous and next words at the same time.

[0054] The present invention introduces the Dropout method and the Early Stopping method to suppress overfitting and improve the generalization ability of the model.

[0055] The present invention can effectively handle the high dimensionality and sparsity problems of word vector space, and shows good performance in multi-dimensional evaluation indicators such as accuracy, macro precision, macro recall and macro F1 value. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is the overall framework of the TCNN-BiGRU vulnerability classification model;

[0057] Figure 2 It is the Skip-gram model structure;

[0058] Figure 3 It is a weighted word vector construction process that combines the N-TF-IDF algorithm with the Word2Vec model;

[0059] Figure 4 It is the TCNN-BiGRU model structure;

[0060] Figure 5 It is the convolution layer and pooling layer of the TextCNN structure;

[0061] Figure 6 It is the internal structure of the GRU model;

[0062] Figure 7 This is a graph showing the relationship between accuracy, training time, and sentence length;

[0063] Figure 8 It is a graph of the loss value changes in the training set and the validation set. DETAILED DESCRIPTION

[0064] The technical solution of the present invention is described in detail below in conjunction with specific implementation methods.

[0065] 1) Data preprocessing

[0066] The present invention uses the data stored in the National Computer Common Vulnerability Database (NVD) of the United States to conduct experimental research on automatic vulnerability classification. Three parts of information, namely CVE-ID, CWE-ID and Description, are extracted from the vulnerability data, wherein CVE-ID represents the number of each vulnerability, such as in the number "CVE-2002-0999", "2002" represents the year when the vulnerability is submitted, and "0999" represents the sequence number of the vulnerability in the vulnerabilities submitted in that year sorted by time; CWE-ID represents the category of the vulnerability, such as "CWE-79", which is the "cross-site scripting" type; Description represents the detailed description text information of the vulnerability, which is an English text. The experiment divides the vulnerability data into a training set, a validation set and a test set at a ratio of 7:2:1, wherein the text set is the detailed description information of the vulnerability, and the label is the vulnerability category. After preprocessing by word segmentation and removal of punctuation, conversion of uppercase letters to lowercase and word form restoration, and removal of stop words and special symbols, a vulnerability is taken as an example, as shown below:

[0067] Vulnerability ID: CVE-2002-0999

[0068] Category: CWE-89

[0069] Description text: Multiple SQL injection vulnerabilities in CARE 2002 beforebeta1.0.02 allow remote attackers to perform unauthorized database operations.

[0070] Preprocessed description text: 'multiple' 'vulnerabilities' 'care' '2002' 'beta' '02' 'perform' 'unauthorized' 'database' 'operations'

[0071] 2) Parameter settings

[0072] The present invention uses the Tensorflow deep learning framework to implement the TCNN-BiGRU model, uses the Skip-gram model of the Word2Vec model under the Gensim tool to train word vectors, and the Early Stopping method is implemented by the Callbacks function under the Keras framework. The parameter settings of the Word2Vec model and the TCNN-BiGRU model are shown in Table 1 and Table 2 respectively.

[0073] Table 1 Parameter settings of Word2Vec model

[0074]

[0075]

[0076] Table 2 Parameter settings of TCNN-BiGRU model

[0077]

[0078] The Skip-gram model consists of a three-layer neural network, including an input layer, a hidden layer, and an output layer. The input layer is the central word x k The one-hot vector of the input layer and the weight matrix between the hidden layer is W V×N , the weight matrix between the hidden layer and the output layer is W' N×V , the output layer obtains the first C vectors y with the largest probability distribution through the activation function 1j ,y 2j ,...,y Cj It is the context vector of the predicted central word, where C is the size of the context window. The model performs gradient optimization training based on the loss function, and continuously updates the weight matrix during the training process. After the training is completed, the weight matrix W V×N This is the desired word embedding matrix, where V is the number of words trained, N is the dimension of the word vector, and the center word x is k The one-hot vector is multiplied by the weight matrix W V×N That is x k The word embedding vector of .

[0079] 3) Hyperparameter Optimization

[0080] (1) Sentence length

[0081] The average length of the vulnerability text after preprocessing is about 20. The input of the convolutional layer requires the same sentence length. The sentence length during fixed-length processing is n. The relationship between the classification accuracy and training time on the validation set and the sentence length n is as follows: Figure 7 shown.

[0082] When n is 120, the classification accuracy is 0.9352 and the training time is about 8241s. When n is greater than 120, the classification accuracy is no longer significantly improved, but the training time is still greatly increased. Considering the comprehensive classification performance and time complexity, this paper sets the sentence length of the convolutional layer input to 120.

[0083] (2) Early Stopping parameters

[0084] The loss values ​​of the training set and the validation set during the training process of the model change as follows Figure 8 As shown in the figure, at the beginning of training, the loss values ​​on the training set and the validation set both decrease with the increase of the number of iterations, but when the number of iterations reaches a certain value, the loss value on the validation set tends to be flat or even starts to increase, and overfitting occurs at this time. Adding the Early Stopping method during the model training process can end the training early, suppress overfitting to a certain extent, and shorten the training time. According to the changes in the loss values ​​of the model on the training set and the validation set, the Early Stopping parameters are set as shown in Table 3.

[0085] Table 3 Early Stopping parameter settings

[0086]

[0087] Using a software vulnerability automatic classification algorithm based on weighted word vectors and neural networks disclosed in this application, the vulnerability records in the internationally recognized U.S. National Computer General Vulnerability Database NVD are used as experimental data. The experiment proves that the vulnerability classification effect is relatively ideal and consistent with the design expectations. The method proposed in the present invention is recorded as N-TF-IDF+Word2Vec+TCNN-BiGRU. Through ablation experiments and comparisons with the following 6 methods, it has good performance in accuracy, macro precision, macro recall, macro F1 value and other indicators, reaching 93.75%, 92.16%, 90.37%, and 91.13% respectively, which are better than all comparison methods in all evaluation indicators. The comparison results with the 6 methods are shown in Table 4.

[0088] Table 4 Comparison of different vulnerability classification methods

[0089]

[0090] According to the comparative experimental results in the above table, the algorithm proposed in this paper is superior to other algorithms in terms of accuracy, macro precision, macro recall, and macro F1 value. Method 1 does not consider the weight of words, and method 2 uses TF-IDF to calculate the weight of words. Comparing method 1 and method 2, the TCNN-BiGRU classifier is also used to prove the effectiveness of the N-TF-IDF method proposed in this paper. In addition, it is noted that the classification performance of method 2 is worse than that of method 1. This is mainly because the traditional TF-IDF algorithm has limitations in calculating word vector weights, making it more suitable for calculating word vector weights for long texts, but the texts in the vulnerability dataset are usually short texts after data preprocessing. The average length of the text in this experiment is about 20 words, so weighting the Word2Vec word vectors by the traditional TF-IDF algorithm does not improve the effectiveness of the Word2Vec word vectors. The N-TF-IDF algorithm proposed in this paper assigns weights to words based on the inter-class distribution. The effectiveness of calculating word vector weights is not limited by the length of the text. Therefore, the N-TF-IDF algorithm proposed in this paper can achieve effective weighting of Word2Vec word vectors.

[0091] The difference between the present invention and method 3 and method 4 is only the classifier. Compared with method 3, the accuracy and macro F1 value (F1 value is a comprehensive evaluation of precision and recall) of the method proposed in the present invention are respectively improved by 16.15% and 20.1%. The performance of method 5 and method 6 are similar and the index values ​​are both around 90%, which proves that both TexCNN and BiGRU classifiers have good classification performance. In the method proposed in the present invention, the combination of the two classifiers makes all the index values ​​higher than 90%, which proves the superiority of the performance of the TCNN-BiGRU classifier in the present invention. In short, the experimental results prove that the method proposed in the present invention is superior to other comparative methods.

Claims

1. A software vulnerability automatic classification method based on weighted word vectors and neural networks, characterized in that The following steps are involved: S1, data preprocessing of vulnerability detailed description text; S2, after the vulnerability text is preprocessed, the weighted word vector is constructed by combining the Word2Vec model and the N-TF-IDF algorithm to complete the text vector representation; The improved word frequency TF is the ratio of the word frequency within the category to the word frequency of all words in the category, as shown in formula (1): Among them, n i,j For word t i The number of occurrences in the jth class, ∑ m n m,j is the number of occurrences of all words in the jth category; the number of category texts is introduced into the improved IDF calculation formula, as shown in formula (2): Where N is the total number of texts in the text set, N i,j is the category j that contains word t i The number of texts, ∑ k N i,k For all categories containing word t i The number of texts; Combining formula (1) and formula (2), the importance of words in different categories of text is shown in formula (3): S3, the TCNN-BiGRU vulnerability classification model based on neural network realizes automatic vulnerability classification. The model consists of four parts, namely input layer, TextCNN structure, BiGRU structure and fully connected layer; the input of the fully connected layer is the final vector obtained by concatenating the feature vectors output by the TextCNN structure and the BiGRU structure.

2. According to claim 1, a method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks is characterized in that Step S1 specifically includes: S11, word segmentation and punctuation removal, text segmentation can be achieved by identifying spaces and punctuation in the vulnerability description text; S12, converting uppercase letters to lowercase and restoring morphology, in this step, different forms of English words are converted into a unified form to facilitate subsequent text processing; S13, remove stop words and special symbols. Stop words include prepositions, articles, and pronouns. Special symbols are characters in the text that have no semantic connection with the context.

3. The method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks according to claim 1 is characterized in that Step S2 specifically includes: S21, trains the word vectors of all words in the text collection through the Word2Vec model, which maps each word into a dense vector in a low-dimensional space, representing the relevant information between words at the semantic level; S22, for the training set, generate a training set vocabulary, which includes all non-repeated words in the training set text, calculate the N-TF-IDF weights of all words in the vocabulary according to formulas (1)-(3), and then form weighted word vectors of all words in the training set vocabulary according to formula (4) combined with Word2Vec word vectors, so as to represent all texts in the training set as a sentence matrix composed of weighted word vectors; S23, for the validation set and the test set, generate a validation set vocabulary and a test set vocabulary respectively. If a word is in the training set vocabulary, its N-TF-IDF weight is set to the N-TF-IDF weight of the word in the training set. If the word is not in the training set vocabulary, its N-TF-IDF weight is set to 0.

5. Then, the weighted word vectors of all words in the vocabulary are formed by combining the Word2Vec word vectors, so that all the texts in the validation set and the test set are represented as sentence matrices composed of weighted word vectors.

4. The method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks according to claim 3 is characterized in that The method to construct the weighted Wrod2Vec word vector is as follows: The matrix M∈R is obtained by the N-TF-IDF algorithm |V|×|K| , where |V| is the number of all non-repeated words in the data set, |K| is the number of categories, that is, the matrix M is the weight of each word in different categories. The maximum value of the weight of each word in all categories is selected as the final weight of the word, and multiplied by its Word2Vec word vector to obtain the final weighted word vector of the word, as shown in formula (4): Weight_Vector(t i )=Word2Vec(t i )*max(N - TF-IDF i,j ) (4) Among them, Weight_Vector(t i ) is the word t i The weighted word vector weight, Word2Vec(t i ) is the word t i The Word2Vec word embedding vector, max() function is used to obtain the word t i The maximum value of weights among all categories.

5. The method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks according to claim 3 is characterized by: The Word2Vec model is a skip-gram model in Word2Vec. The Skip-gram model predicts the context based on the central word. The input is the one-hot vector of the central word, and the output is the prediction of the context of the central word.

6. The method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks according to claim 1, characterized in that Step S3 specifically includes: S31, the input of the input layer is a sentence matrix represented by a word embedding vector; since the input of the convolutional layer requires the same sentence length, it is necessary to perform fixed-length processing, set the length to n, truncate if it exceeds, and fill in 0 if it is less than, the matrix of the input layer is recorded as W n×d , where n is the number of words in the sentence and d is the dimension of the word vector; S32, the TextCNN structure consists of convolutional layers and pooling layers; in the convolutional layer, convolutional kernels of different sizes are applied to extract features from the input matrix, and the convolution kernel is defined as ω∈R h×d , where h is the height of the convolution kernel, which is a hyperparameter and is usually set to 2 to 5, and d is the width of the convolution kernel, which is the same as the dimension of the word vector. The convolution kernel performs a convolution operation on each possible window in the sentence vector matrix to generate a feature map S j ∈R n-h+1 , where n is the number of words in the sentence, and convolution kernels of different sizes will generate feature maps of different sizes; S33, using Bidirectional Gated Recurrent Unit, BiGRU, BiGRU applies GRU twice in opposite directions, that is, at each time step t, the input data is passed to the forward hidden layer and the backward hidden layer at the same time. Finally, the output of BiGRU is obtained by concatenating the feature vectors output by the two unidirectional GRUs; The core structure of GRU includes two gate structures: reset gate and update gate; the specific calculation formula of GRU internal structure is as follows: z t =σ(W z ·[h t-1 ,x t ]) (5) r t =σ(W r ·[h t-1 ,x t ]) (6) Among them, r t Represents the gate value of the reset gate, z t represents the gate value of the update gate, h t-1 is the output of the hidden layer at time t-1, x t is the input at time t, h t is the output of the hidden layer at time t, Represents the candidate value of the hidden layer node at time t, three W z , W r , W represents the linear transformation operation, * represents the matrix element multiplication, σ represents the sigmoid activation function, y t Indicates output; and are the states of the forward and backward hidden layers at time step t, respectively. and are the states of the forward and backward hidden layers at time step t-1, respectively, and h t is the final output of the BiGRU layer; The feature vector output by the BiGRU layer is input into the pooling layer, and the maximum pooling operation is used to continue to extract the feature vector with the strongest representative ability, thereby reducing the dimension of the output vector; S34, in order to suppress the overfitting problem of the model, a Dropout layer was added between the pooling layer and the fully connected layer of the TCNN-BiGRU model, and the Early Stopping method was added during the training process of the model; S35, the fully connected layer is similar to the hidden layer in the traditional neural network. Each neuron in it is fully connected to all neurons in the previous layer, integrating local feature maps to obtain global feature information, and sending the output value to the Softmax classifier; the Softmax activation function operates on the vector, maps the value in the vector to a value between (0, 1), and obtains the probability that the sentence belongs to each category. The category with the largest probability value is the predicted classification result; the calculation formula of the Softmax function is as follows: Among them, p(y i s) is the probability that the sentence belongs to the i-th category, y i is the i-th value of the vector, and c is the number of categories.

7. The method for automatic classification of software vulnerabilities based on weighted word vectors and neural networks according to claim 6, characterized in that: Applying convolution kernels of different sizes in the convolution layer generates feature maps of different sizes. It is necessary to perform a pooling operation on each feature map to obtain features of the same dimension. The maximum pooling operation is used to pool the output of the convolution layer, and the maximum value of each feature map is extracted to represent the feature, thereby capturing the most important features. Through the maximum pooling operation, each convolution kernel obtains a feature value, and the pooling operation is performed on all convolution kernels, and then they are concatenated to obtain the final feature vector of the pooling layer.

Images