Software vulnerability detection method based on pre-training model feature fusion

Abstract

The invention discloses a pre-training model feature fusion-based software vulnerability detection method, which comprises the following steps of: 1) acquiring a software code, and generating a training set of a model; the software code is an original code of software; 2) segmenting and preprocessing the obtained software code to be detected; (3) after the input data are segmented and preprocessed, software codes are converted into Tokens sequences through byte pair coding (BPE); 4) feature fusion; 5) inputting the feature fusion vector representation into a code vulnerability detection model based on deep learning, and training the model; 6, the model is tested, the verified model is used for vulnerability detection, and whether a code segment contains vulnerabilities or not is obtained according to a detection result. According to the software vulnerability detection method based on pre-training model feature fusion, the model detection precision is guaranteed, and meanwhile the model detection performance is considered.

Description

Technical Field

[0001] This invention relates to information security technology, and in particular to a software vulnerability detection method based on feature fusion of a pre-trained model. Background Technology

[0002] The diversity and complexity of software result in insufficient generalization ability of existing rule-based and deep learning models. Rule-based methods struggle to enumerate all vulnerability types and are costly to maintain. Data-driven deep learning models are limited by the scale and diversity of training data and are susceptible to data bias. While large-scale models offer strong vulnerability generalization, they suffer from low detection efficiency and high costs, making them unsuitable for the needs of vulnerability detection in industrial fields. Therefore, improving the generalization ability of models while maintaining performance has become a pressing problem for vulnerability detection. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a software vulnerability detection method based on feature fusion of a pre-trained model, which addresses the shortcomings of the existing technology.

[0004] The technical solution adopted by this invention to solve its technical problem is: a software vulnerability detection method based on pre-trained model feature fusion, comprising the following steps: 1) Obtain the software code to form a training set; the software code is the original code of the software; 2) Code parsing and preprocessing; 2.1) For the input training set code, parse the original code into an abstract syntax tree form; Use Tree-sitter to parse the source code and build an Abstract Syntax Tree (AST); 2.2) Parse the software code into function-level code granularity; Traverse the Abstract Syntax Tree (AST) and identify function definition nodes to decompose the entire code into multiple independent functions, thus obtaining code segmentation at the function-level code granularity. 2.3) Traverse the generated abstract syntax tree to obtain syntax tree nodes that may cause code vulnerabilities. For data with vulnerabilities, retain the vulnerable functions according to the number of vulnerable lines in the tags. 2.4) Reduce redundant information in software code data; Finally, redundant information in the code data is reduced by removing comments, standardizing spaces and newlines; 3) The software code data processed in step 2) is segmented, and byte pair encoding (BPE) is used to convert the software code into a sequence of tokens; The conversion method is as follows: Initially, each character in the software code is treated as a separate token. In each iteration, the algorithm counts the frequency of all adjacent character pairs and selects the character pair with the highest frequency to merge into a new token. This process will be repeated until the preset vocabulary size or number of merges is reached. 4) Feature fusion The embedding vectors of the token sequence are extracted using both the encoder-only architecture pre-trained model and the decoder-only architecture pre-trained model. The feature fusion vector is obtained by concatenating the embedding vectors from two different sources; 5) Input the feature fusion vector representation into the deep learning-based code vulnerability detection model for model training; 6) Test the model, use the verified model to perform vulnerability detection, and determine whether the code segment contains vulnerabilities based on the detection results.

[0005] According to the above scheme, in step 1), the training set is generated based on the synthetic training set sample data of the large language model; specifically as follows: 1.1) Expand the training dataset with additional samples; Based on the original code, the large language model is guided to generate data by using zero-shot and few-shot prompts to obtain expanded samples; 1.2) Sample filtering; The samples are filtered to obtain expanded samples that meet the specifications; 1.3) Sample variation; 1.3.1) Slice the extended sample code that conforms to the specifications and retain the code statements related to the vulnerability; 1.3.2) After retaining the code statements that contain relevant vulnerability characteristics, the statements that are not related to the vulnerability are mutated to enhance the diversity of the dataset samples; 1.4) Mix the original samples, the standardized augmented samples, and the variant samples to obtain a class-balanced synthetic training set of sample data.

[0006] According to the above scheme, the sample filtering includes the following steps: 1.1.1) Code Similarity Detection The Simian tool is used to detect code data generated by large models and remove data samples with similarity higher than a set threshold. When configuring Simian, the threshold is set to 0.9 to eliminate similar code. 1.1.2) Sample Filtering By analyzing the differences between the three detection tools, pseudo-labels are applied to the data samples to be tested, and then compared and corrected with the original labels; the details are as follows: (a) For a specific vulnerability type, a small number of labeled dataset samples are used to detect vulnerabilities using three vulnerability detection tools to obtain the accuracy of each vulnerability type, which is used as the confidence level of the detection tool in predicting a specific vulnerability type for an unknown sample. The three vulnerability detection tools are Fortify (a static analysis tool), JBMC (a bounded model inspection tool for Java code), and AIDetectVul (a tool for detecting vulnerabilities). (b) Assign weights to the three detection tools for different vulnerability types; (c) After obtaining the weight of each category, the generated data samples are detected using three detection tools, and the weighted sum score of each category is calculated and predicted. Then, the category with the highest score in the results is taken as the final detection result of the test sample. (d) Compare the detection results with the original label categories, correct any erroneous annotations that may exist in the generated data samples, and obtain a dataset that conforms to the standard.

[0007] The beneficial effects of this invention are: 1. This invention proposes a software vulnerability detection method based on feature fusion of a pre-trained model, which optimizes both the vulnerability representation level and the model level, ensuring the accuracy of model detection while also taking into account the model detection performance (i.e., training cost, time cost, etc.).

[0008] 2. This invention improves the sample training set, and the constructed hybrid dataset can enhance the model's generalization ability. Attached Figure Description

[0009] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of a method according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the feature fusion principle of an embodiment of the present invention; Figure 3 This is a model structure diagram of an embodiment of the present invention. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0011] like Figure 1As shown, a software vulnerability detection method based on feature fusion of a pre-trained model includes the following steps: 1) Obtain the labeled software code to form a training set; the software code is the original code of the software; Since the original code may have insufficient distribution in terms of the number and types of vulnerabilities, in order to better train the model, this embodiment uses synthetic training set sample data based on a large language model to expand the training dataset. 1.1) Guide the large language model to generate data and expand the training dataset by using zero-shot and few-shot prompts; 1.2) Filter the expanded samples; The samples are filtered to obtain expanded samples that meet the specifications; Sample filtering includes the following steps: 1.1.1) Code Similarity Detection The Simian tool is used to detect code data generated by large models and remove data samples with similarity exceeding a set threshold. When configuring Simian, the threshold is set to 0.9 to eliminate similar code, as a threshold that is too low will significantly eliminate code samples valuable for machine learning. Even small changes can improve the training process by providing different representations of the same vulnerability type, allowing the model to effectively identify that type in other scenarios. 1.1.2) Sample Filtering After removing highly similar samples, the quality of the dataset labels is verified and filtered to obtain a high-quality dataset that meets the training requirements. At the same time, for the insufficient number of vulnerability types, large models are used to generate data to supplement them.

[0012] We selected Fortify, a state-of-the-art static analysis tool in the vulnerability detection field, JBMC, a bounded model inspection tool for Java code, and AIDetectVul as evaluation tools, and combined their detection results through an integration strategy. Each detection tool focuses on different vulnerability types. Fortify, a static code analysis tool, has a very rich built-in security rule base, supporting 1657 vulnerability types, and focuses on SQL injection, cross-site scripting (XSS), buffer overflow, and other vulnerability types. Our team also developed specific rules for the specific vulnerability types in the enterprise private data used in this paper, resulting in high accuracy for Fortify's vulnerability type detection. JBMC, developed based on the CPROVER framework, can detect abnormal situations in Java program execution, focusing on null pointer exceptions, array out-of-bounds errors, arithmetic overflow, and other vulnerability types. AIDetectVul achieved an F1 score of 88% when trained on the enterprise private dataset, thus its detection results are also meaningful.

[0013] This embodiment employs an integrated detection strategy based on a weighted probability voting mechanism. This strategy primarily utilizes the differences between the three detection tools to apply pseudo-labels to the data samples to be detected, followed by comparison and error correction with the original labels. The specific integration strategy is as follows: (a) For a specific vulnerability type, a small number of labeled dataset samples are used to detect each vulnerability type using three detection tools to obtain the accuracy of each vulnerability type, which serves as the confidence level of the detection tool in predicting a specific vulnerability type for an unknown sample; (b) Assign weights to the three detection tools for different vulnerability types; (c) After obtaining the weight of each category, the generated data samples are detected using three detection tools, and the weighted sum score of each category is calculated and predicted. Then, the category with the highest score in the results is taken as the final detection result of the test sample. (d) Compare the detection results with the original labels, correct any erroneous annotations that may exist in the generated data samples, and obtain a dataset that conforms to the standard.

[0014] 1.3) Sample variation; 1.3.1) Slice the extended sample code that conforms to the specifications and retain the code statements related to the vulnerability; 1.3.2) After retaining the code statements that contain relevant vulnerability characteristics, the statements that are not related to the vulnerability are mutated to enhance the diversity of the dataset samples; 1.4) The original sample conforms to the specifications. We combine the generated data and the data-enhanced mutated data from these three sources to obtain a class-balanced synthetic training set sample data.

[0015] 2) Code parsing and preprocessing; 2.1) For the input raw code, parse the raw code into an abstract syntax tree form; In this embodiment, Tree-sitter is used to parse the source code and construct an Abstract Syntax Tree (AST); 2.2) Parse the software code into function-level code granularity; Traverse the Abstract Syntax Tree (AST) and identify function definition nodes to decompose the entire code into multiple independent functions, thus obtaining code segmentation at the function-level code granularity. 2.3) Traverse the generated abstract syntax tree to obtain syntax tree nodes that may cause code vulnerabilities. For data with vulnerabilities, retain the functions with vulnerabilities based on the number of lines with vulnerabilities in the tags. 2.4) Reduce redundant information in the data In this embodiment, redundant information in the data is reduced by removing comments, standardizing spaces and newlines, etc. 3) After segmenting and preprocessing the input data, the software code is converted into a sequence of tokens using byte pair encoding (BPE); The conversion method is as follows: Initially, each character in the software code is treated as a separate token. In each iteration, the algorithm counts the frequency of all adjacent character pairs and selects the character pair with the highest frequency to merge into a new token. This process will be repeated until the preset vocabulary size or number of merges is reached. 4) Feature fusion The embedding vectors of the token sequence are extracted using both the encoder-only architecture pre-trained model and the decoder-only architecture pre-trained model. The feature fusion vector is obtained by concatenating the embedding vectors from two different sources; What are the commonly used pre-trained models with encoder-only architecture and decoder-only architecture, and can you provide examples of the extracted embedding vector forms?

[0016] CodeBERT, as a BERT-based model, excels in understanding the context and structure of code. It can capture the semantic and syntactic features of code, and numerous studies have demonstrated its effectiveness in code understanding tasks.

[0017] The decoder-only model focuses more on the potential patterns in the code generation process. It provides dynamic features of the code from the decoding perspective, capturing not only the local structure of the code, such as variable names and function names, but also the contextual information and programming patterns of the code. This gives it a unique advantage in understanding the code generation logic and potential errors.

[0018] These two models extract code features from different perspectives and are naturally complementary. The encoder-only model excels at capturing the static semantic information of the code, while the decoder-only model excels at capturing the dynamic generation patterns and contextual information of the code. By fusing these two types of embedding vectors, the features of the code can be captured more comprehensively, enhancing feature diversity and thus improving the model's generalization ability. The code embedding vector synthesis process is as follows: Figure 2 As shown.

[0019] Taking CodeBERT as an example, CodeBERT contains 12 encoders. Each encoder outputs a tensor for the input token sequence. The output of the previous encoder becomes the input of the next encoder, and so on, until the output of the 12th encoder becomes the output of the entire model. This progressive data progression by encoder achieves multi-layered feature extraction and refinement. The output of the last [CLS] identifier is then used as the source code feature vector, which contains the semantic representation of the entire source code. First, a pre-trained model is used to encode each token using both word and position, generating corresponding word vectors and positional encoding vectors. Word vectors capture meaningful relationships between a given token and other tokens, while positional encoding vectors represent the position of a given token in the input sequence. These vectors are then merged into a code embedding vector (i.e., a code feature vector), which not only contains semantic information but also rich contextual information.

[0020] Through the above steps, the encoder model generates a 1×768-dimensional code embedding, and the decoder model generates a 1×1024-dimensional code embedding. Finally, these two vectors are concatenated to form the final fused feature.

[0021] 5) Input the feature fusion vector representation into the deep learning-based code vulnerability detection model to train the model; like Figure 3 This invention uses a Transformer architecture model with a 6-layer encoder structure as a classifier, where each encoder layer includes a multi-head attention layer and a feedforward neural network. The training process for the Transformer model is as follows: 5.1) Self-attention mechanism; a. First, input matrix X (code feature vector) is compared with three different training weight matrices W. Q W K and W V Multiplying them transforms them into a Query matrix Q, a Key matrix K, and a Value matrix V; ; b. Calculate the dot product of the query matrix Q and the key matrix K to obtain the attention score matrix S; ; c. To stabilize the training process, divide the attention score matrix S by the dimension of the key vector. The square root of the result yields the normalized attention score;

[0022] ; d. After performing a Softmax operation on the normalized attention score matrix S', multiply it by the value matrix V to obtain the weighted value matrix, which is the output of the self-attention mechanism; ; 2) Multi-head attention mechanism; Enhance the model's expressive power by running multiple attention mechanisms in parallel; The input sequence is subjected to multiple different linear transformations to obtain multiple sets of Query, Key, and Value matrices. The attention weights are calculated and multiplied by the Value matrix to obtain a weighted Value matrix. Then, the weighted matrices of all heads are concatenated and subjected to a linear transformation to obtain the final output of the multi-head attention. The calculation formula is as follows:

[0023] Among them, W O It is the output weight matrix; 3) Perform residual connections and layer normalization on the output of the multi-head attention layer; This operation helps gradient flow, alleviates the gradient vanishing problem, stabilizes the training process, and speeds up convergence. 4) Input the normalized result into the feedforward neural network for further data processing, and then perform residual connections and layer normalization again to ensure the integrity of information and the stability of training.

[0024] 5.5) Steps 5.1) to 5.4) are the operations of the input sequence through an encoder layer. The output after processing by an encoder layer is sent to the next encoder layer. This process is repeated, and finally the representation of the encoder output is meaningful.

[0025] 5.6) After pooling the output, the feature dimensions are mapped to a single output node through a fully connected layer for the final classification task.

[0026] 6) Test the model, use the verified model to perform vulnerability detection, and determine whether the code segment contains vulnerabilities based on the detection results.

[0027] 2. Experimental Design and Analysis 2.1 Dataset Due to the privacy of corporate private data, the generalizability and reproducibility of research results may be affected. Therefore, this experiment uses both corporate private datasets and open-source public datasets to verify the method, ensuring its effectiveness in a specific domain and verifying its generalizability in a wider range of environments.

[0028] The enterprise's private dataset comes from a listed financial company. After the data is initially scanned by a rules engine, it undergoes a rigorous code review by a security expert group before being labeled. Furthermore, the company has strict code writing standards, so the dataset itself is of high quality and accuracy.

[0029] The open-source public dataset used is the real-world DiverseVul dataset, which, compared to other open-source datasets, covers a wider range of CWE vulnerability types and has a larger number of positive and negative samples. Specific information about the two datasets is shown in Table 2.

[0030] Table 2 Dataset Statistics

[0031] To comprehensively evaluate the model's detection accuracy and generalization ability, the detection accuracy was evaluated using data from the same project source, while the generalization ability was evaluated using data from different project sources. Therefore, all data from a specific project was extracted as generalization test data, and all data from the remaining projects were divided into training and test sets in a 9:1 ratio.

[0032] 2.2 Experimental environment (as shown in Table 3) Table 3 Experimental Environment

[0033] 2.3 Evaluation Indicators With industrial vulnerability detection as the research background, it is necessary to comprehensively evaluate the performance of the model in multiple aspects. Therefore, the model performance, detection accuracy and generalization ability are evaluated separately. Experimental comparison and evaluation are conducted below.

[0034] 2.3.1 Model Performance Performance is primarily determined by the model's training and testing time. This metric reflects the model's detection efficiency and is particularly important in industrial vulnerability detection.

[0035] (1) Training time: The time required for the model to converge on the training data.

[0036] (2) Test time: The time required for the model to test a single test sample.

[0037] 2.3.2 Model Accuracy Detection accuracy is evaluated by training and testing on data from the same project source. This metric reflects the model's ability to identify known vulnerabilities.

[0038] (1) Accuracy: Accuracy is the most intuitive performance indicator, representing the proportion of correctly detected samples out of the total number of samples. The calculation formula is as follows:

[0039] In this context, TP (True Positives) represents true positives, TN (True Negatives) represents true negatives, FP (False Positives) represents false positives, and FN (False Negatives) represents false negatives.

[0040] (2) Precision: Precision focuses on the proportion of samples that the model detects as positive, but which are actually positive. The calculation formula is as follows:

[0041] The higher the accuracy, the more reliable the model is in detecting positive examples, that is, the lower the false alarm rate.

[0042] (3) Recall: Recall, also known as sensitivity or TruePositive Rate (TPR), refers to the proportion of all true positives that are correctly detected as positive. The calculation formula is as follows:

[0043] A higher recall rate indicates that the model can capture more positive examples, meaning a lower false negative rate.

[0044] (4) F1 Score: The F1 score is the harmonic mean of precision and recall, used to comprehensively consider the performance of precision and recall. The calculation formula is as follows:

[0045] The harmonic mean has less impact on extreme values ​​during calculation, so the F1 score can balance precision and recall, avoiding one metric being too high while the other is too low. In imbalanced datasets, the ratio of positive to negative samples can be very disparate, and using precision or recall alone may lead to a biased evaluation of model performance. The F1 score, by comprehensively considering both metrics, better reflects the overall capability of a model.

[0046] 2.3.3 Model Generalization Generalization is assessed by performing K-fold cross-validation using data from different project sources, calculating mean precision, recall, and F1 score. This metric reflects the model's ability to identify unknown vulnerabilities, i.e., whether the model can adapt to new, unseen data.

[0047] K-Fold Cross-Validation is a statistical method for evaluating the generalization ability of a model. It involves splitting the dataset into K mutually exclusive subsets of similar size, then performing K training and validation iterations. Each iteration selects a different subset as the validation set, and the rest are used as the training set. The average of the model's performance metrics (such as precision, recall, and F1 score) over the k iterations is then calculated as the evaluation result for the model's generalization ability.

[0048] 2.4 Experimental Results In this embodiment, experiments were conducted using enterprise private datasets and open-source public datasets, and a comprehensive evaluation was performed. The experimental setup for comparison was as follows: (1) Fortify: a popular rule-based static analysis tool for software; (2) LineVul: a row-level vulnerability detection model based on Transformer, which has the performance of a domain-specific state-of-the-art model; (3) CodeT5 Small, which has stronger vulnerability detection capabilities in models with small parameter scales and has a similar number of parameters to the pre-trained model used in this technical solution; (4) Qwen1.5-110B: a large model with hundreds of billions of parameters and strong generalization capabilities. The training parameters of the Transformer classification model used in this technical solution are shown in Table 4.

[0049] Table 4 Training parameters of the Transformer model

[0050] To verify the results of this study, the model of this invention was compared with different models, and the results are shown in Table 5.

[0051] Table 5. Validation Experiment Results of Enterprise Private Dataset

[0052] Based on the above results, the following conclusions can be drawn: 1) Feature extraction based on the decoder-only architecture model outperforms the currently widely used encoder-only models in terms of detection accuracy and generalization. 2) Compared with the current state-of-the-art model LineVul, the model of this invention has similar detection accuracy on the same project data, but its generalization on unseen data is improved by 3.75%. 3) Through the comparison of the results of the last three experiments (ablation experiments), the feature fusion strategy proposed in this invention outperforms models using encoder-only features and decoder-only features in both detection accuracy and generalization ability, proving that feature fusion of different architecture pre-trained models can enhance feature diversity, thereby improving the detection accuracy and generalization of the model. 4) Qwen1.5-110B demonstrates strong generalization ability, achieving an accuracy of 96% in detecting vulnerable samples, but its false positive rate is still relatively high.

[0053] The generalization experiment of the model of this invention was carried out on an unseen dataset to analyze its ability to detect different types of vulnerabilities. The experimental results are shown in Table 6.

[0054] Table 6 Generalization Test Results for Each Vulnerability Type

[0055] The above results indicate that the model of this invention has a high learning rate for negative samples. It not only maintains high accuracy for vulnerability types already learned during training on unseen datasets, but also demonstrates some detection capability for unseen vulnerability types (such as CWE327, CWE328, and CWE643).

[0056] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A software vulnerability detection method based on feature fusion of a pre-trained model, characterized in that, Includes the following steps: 1) Obtain the software code and generate the training set for the model; the software code is the original code of the software; 2) The acquired software code to be detected is segmented and preprocessed; 3) After segmenting and preprocessing the input data, the software code is converted into a sequence of tokens using byte pair encoding (BPE). 4) Feature fusion Embedding vectors of the token sequence are extracted using pre-trained models with encoder-only and decoder-only architectures, respectively. The feature fusion vector is obtained by concatenating the embedding vectors from two different sources; 5) Input the feature fusion vector representation into the deep learning-based code vulnerability detection model to train the model; 6) Test the model, use the verified model to perform vulnerability detection, and determine whether the code segment contains vulnerabilities based on the detection results.

2. The software vulnerability detection method based on pre-trained model feature fusion according to claim 1, characterized in that, In step 1), the training set is generated based on the synthetic training set sample data of the large language model; specifically as follows: 1.1) Expand the training dataset with additional samples; Based on the original code, the large language model is guided to generate data by using zero-shot and few-shot prompts to obtain expanded samples; 1.2) Sample filtering; The samples are filtered to obtain expanded samples that meet the specifications; 1.3) Sample variation; 1.3.1) Slice the extended sample code that conforms to the specifications and retain the code statements related to the vulnerability; 1.3.2) After retaining the code statements that contain relevant vulnerability characteristics, the statements that are not related to the vulnerability are mutated to enhance the diversity of the dataset samples; 1.4) Mix the original samples, the standardized augmented samples, and the variant samples to obtain a class-balanced synthetic training set of sample data.

3. The software vulnerability detection method based on pre-trained model feature fusion according to claim 2, characterized in that, The sample filtering Includes the following steps: 1.1.1) Code Similarity Detection The Simian tool is used to detect code data generated by large models and remove data samples with similarity higher than a set threshold. When configuring Simian, the threshold is set to 0.9 to eliminate similar code. 1.1.2) Sample Filtering By analyzing the differences between the three detection tools, pseudo-labels are applied to the data samples to be tested, and then compared and corrected with the original labels; the details are as follows: (a) For a specific vulnerability type, a small number of labeled dataset samples are used to detect vulnerabilities using three vulnerability detection tools to obtain the accuracy of each vulnerability type, which is used as the confidence level of the detection tool in predicting a specific vulnerability type for an unknown sample. The three vulnerability detection tools are Fortify (a static analysis tool), JBMC (a bounded model inspection tool for Java code), and AIDetectVul (a tool for detecting vulnerabilities). (b) Assign weights to the three detection tools for different vulnerability types; (c) After obtaining the weight of each category, the generated data samples are detected using three detection tools, and the weighted sum score of each category is calculated and predicted. Then, the category with the highest score in the results is taken as the final detection result of the test sample. (d) Compare the detection results with the original label categories, correct any erroneous annotations that may exist in the generated data samples, and obtain a dataset that conforms to the standard.

4. The software vulnerability detection method based on pre-trained model feature fusion according to claim 1, characterized in that, In step 2), segmenting the acquired software code to be detected includes: For the input raw code, parse the raw code and construct an Abstract Syntax Tree (AST); Parse software code into function-level code granularity; Traverse the Abstract Syntax Tree (AST) and identify function definition nodes to decompose the entire code into multiple independent functions, thus obtaining code segmentation at the function-level code granularity. 2.3) Traverse the generated abstract syntax tree to obtain syntax tree nodes that may cause code vulnerabilities. For data with vulnerabilities, retain the vulnerable functions according to the number of vulnerable lines in the tag.

5. The software vulnerability detection method based on pre-trained model feature fusion according to claim 1, characterized in that, In step 3), the conversion method is as follows: Initially, each character in the software code is regarded as a separate token. In each iteration, the algorithm will count the frequency of occurrence of all adjacent character pairs and select the character pair with the highest frequency to merge into a new token. This process will be repeated until the preset vocabulary size or number of merges is reached.

6. The software vulnerability detection method based on pre-trained model feature fusion according to claim 1, characterized in that, In step 5), a Transformer architecture model with a 6-layer encoder structure is used as the code vulnerability detection model.

7. The software vulnerability detection method based on pre-trained model feature fusion according to claim 6, characterized in that, In step 5), the training method is as follows: The training process is as follows: 5.1) Self-attention mechanism; 5.1.1) First, the input code feature vector matrix X is compared with three different training weight matrices W. Q W K and W V Multiplying them transforms them into a Query matrix Q, a Key matrix K, and a Value matrix V; ； 5.1.2) Calculate the dot product of the Query matrix Q and the Key matrix K to obtain the attention score matrix S; ； 5.1.3) To stabilize the training process, divide the attention score matrix S by the dimension of the key vector. The square root of the result yields the normalized attention score; ； 5.1.4) After performing a Softmax operation on the normalized attention score matrix S', multiply it by the value matrix V to obtain the weighted value matrix, which is the output of the self-attention mechanism; ； 5.2) Multi-head attention mechanism; Enhance the model's expressive power by running multiple attention mechanisms in parallel; The input sequence is subjected to multiple different linear transformations to obtain multiple sets of Query, Key, and Value matrices. Attention weights are calculated and multiplied by the Value matrix to obtain a weighted Value matrix. Then, the weighted matrix of all heads is calculated. After concatenation, a linear transformation is applied to obtain the final output of the multi-head attention algorithm. The calculation formula is as follows: Among them, W O It is the output weight matrix; 5.3) Perform residual connections and layer normalization on the output of the multi-head attention layer; 5.4) The normalization result is input into the feedforward neural network for further data processing, followed by residual connection and layer normalization again; 5.5) Steps 5.1) to 5.4) are the operations of the input sequence through an encoder layer. The output after processing by an encoder layer is sent to the next encoder layer. This process is repeated to obtain the final encoder output. 5.6) After pooling the output, the feature dimensions are mapped to a single output node through a fully connected layer for the final classification task.

8. The software vulnerability detection method based on pre-trained model feature fusion according to claim 1, characterized in that, In step 2), the preprocessing of the acquired software code to be detected includes: reducing redundant information in the software code data; Reduce redundant information in the data by removing comments, standardizing spaces and line breaks.

9. An electronic device, characterized in that, include: One or more processors; as well as Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1 to 8.

Images