Vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention
By combining multimodal feature fusion of vulnerability function code and descriptive text with cross-task attention mechanism, the problem of single data source and neglect of task correlation in existing technologies is solved, and a more comprehensive vulnerability assessment effect is achieved.
Patent Information
- Application Number
- CN202511812613.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-04
- Publication Date
- 2026-03-17
AI Technical Summary
Existing vulnerability assessment methods typically rely on a single data source, which fails to fully capture the potential features required for vulnerability assessment and fails to effectively utilize the correlation between assessment tasks.
We employ a vulnerability assessment method based on multimodal feature fusion of code and text and cross-task attention. This method preprocesses the vulnerable function code and description text to extract features, and then uses a cross-task attention mechanism to fuse multi-task related features for vulnerability assessment.
It enables more comprehensive vulnerability assessment, capturing detailed information at the code level and background information at the text level, promoting interaction between tasks, and improving the accuracy and comprehensiveness of the assessment.
Smart Images
Figure CN121682840A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of function-level vulnerability assessment technology, specifically a vulnerability assessment method based on the fusion of multimodal features of code and text and cross-task attention. Background Technology
[0002] Software vulnerability assessment is a process that takes vulnerability data as input, outputs vulnerability severity scores, and presents vulnerabilities to developers in descending order of severity scores. Current vulnerability assessment standards primarily use the Common Vulnerability Scoring System (CVSS). It includes a series of assessment metrics, such as access vector, access complexity, authentication, confidentiality impact, integrity impact, and availability impact, to evaluate different aspects of vulnerabilities. There is also a comprehensive assessment metric: severity. Due to the limitations of software quality assurance resources, software vendors and security experts cannot address all vulnerabilities in a timely manner. Therefore, based on assessment metrics and the needs of different scenarios, vulnerabilities requiring priority remediation are selected and assigned to developers for fix.
[0003] However, existing methods often rely on a single data source, such as vulnerable code or vulnerability description text, for vulnerability assessment tasks. Using only vulnerability description text is problematic because this type of information is typically short and contains limited details, potentially resulting in insufficient information relevant to the seven vulnerability assessment tasks and inaccurate predictions. While using only vulnerable code can capture some implementation details at the code level, information related to the context and external conditions in which the vulnerability occurs is difficult to obtain at this level alone. Using a single data source leads to an inability to comprehensively capture the potential features required for vulnerability assessment.
[0004] Existing vulnerability assessment methods often treat the seven CVSS assessment tasks as independent classification tasks, failing to consider the correlations between them. Previous research has also confirmed the correlations between different basic metrics in CVSS assessments. Although some methods employ multi-task learning for the seven vulnerability assessment tasks, the feature vectors of different CVSS metrics do not interact, thus failing to capture the correlations between the tasks effectively. Summary of the Invention
[0005] The purpose of this invention is to overcome the shortcomings of existing technologies and propose a vulnerability assessment method based on the fusion of multimodal features from code and text and cross-task attention. This method effectively alleviates the problems of previous methods that only use a single data source and ignore the correlation between assessment tasks.
[0006] To achieve the above objectives, the technical solution specifically adopted by the present invention is as follows:
[0007] A vulnerability assessment method based on multimodal feature fusion of code and text and cross-task attention includes the following steps:
[0008] Step 1: Obtain and preprocess vulnerability data, which includes vulnerability function code and vulnerability description text;
[0009] The preprocessing of the vulnerable function code includes: converting the code into a control flow graph and generating multiple execution paths from the control flow graph;
[0010] Preprocessing of the vulnerability description text includes: extracting vulnerability elements from the vulnerability description text;
[0011] Step 2: Extract features from the preprocessed vulnerability function code and vulnerability description text, and use a cross-task attention mechanism to fuse multi-task related features to obtain code fusion features and text fusion features;
[0012] Step 3: Based on the extracted code fusion features and text fusion features, obtain the predicted values through classifiers, perform weighted fusion, and output the vulnerability assessment results.
[0013] Preferably, the specific steps for converting the code into a control flow graph include:
[0014] Use regular expressions to remove blank lines and comments from the code, and number each line of code.
[0015] Use a syntax parsing tool to parse the code into an abstract syntax tree;
[0016] Construct a control flow graph based on an abstract syntax tree.
[0017] As a preferred option, only statement nodes are retained when constructing the control flow graph.
[0018] Preferably, generating multiple execution paths from the control flow graph includes:
[0019] Traverse the path from the entry node to the exit node;
[0020] Choose multiple paths, where the first m-1 paths are short and distinct, and the m-th path covers the code node.
[0021] Preferably, the method for extracting vulnerable elements is as follows:
[0022] Perform word segmentation, stop word removal, and word form restoration on the text;
[0023] Vulnerability elements are extracted using a pre-trained BERT-MRC model.
[0024] Preferably, in step 2:
[0025] Feature extraction of the code execution path includes: encoding using the CodeBERT model, extracting initial features for the seven evaluation tasks through seven independent convolutional neural networks, and finally fusing multi-task features using a cross-task attention mechanism to obtain code fusion features;
[0026] Feature extraction of text vulnerability elements includes: initial embedding using the GloVe word embedding model, then initial features for seven evaluation tasks are extracted using seven independent convolutional neural networks, and finally, multi-task features are fused using a cross-task attention mechanism to obtain text fusion features.
[0027] Preferably, the cross-task attention mechanism includes a relevance-guided attention mechanism and a self-attention mechanism.
[0028] Preferably, in step 2, the relevance-guided attention mechanism utilizes cross-interrogation of spatial relevance to guide the extraction of contribution features of secondary tasks to primary tasks; and a self-attention mechanism self-discovers complementary features of the primary tasks. For any two evaluation tasks, their initial features are input into the above module to obtain contribution features and complementary features. Connecting these two features yields the interaction features between the two evaluation tasks. Ultimately, the fused features of each evaluation task are derived from the combination of the initial features of the current task and the interaction features with other tasks.
[0029] Preferably, in step 3:
[0030] For each evaluation task, its code fusion features and text fusion features are processed by a multilayer perceptron classifier to obtain their respective logits;
[0031] For each evaluation task, the logits of the code and text are weighted and summed, and then an activation function is used to obtain the probability distribution of different labels for that evaluation task, and finally the CVSS evaluation index is determined.
[0032] Preferably, the CVSS evaluation metrics include: access vector, access complexity, authentication, confidentiality impact, integrity impact, availability impact, and severity.
[0033] Preferably, the method is used for function-level vulnerability severity assessment, and based on the assessment results, vulnerabilities that need to be patched are prioritized.
[0034] This invention has the following characteristics and beneficial effects:
[0035] In the field of vulnerability assessment, since each assessment task considers different aspects, more comprehensive information is needed to aid in the assessment. This invention combines the code information of the vulnerable function with the vulnerability description text information. It can capture detailed information about the vulnerability implementation at the code level, and information related to the background and external conditions of the vulnerability at the text level, thus more comprehensively capturing the potential features required for vulnerability assessment. Compared to treating each assessment task as an independent task, this invention introduces a cross-task attention mechanism, which can promote interaction between different tasks, thereby better capturing the correlation between assessment tasks. Attached Figure Description
[0036] Figure 1 This is a schematic diagram of the overall framework of the vulnerability assessment method based on code and text multimodal feature fusion and cross-task attention in an embodiment of the present invention.
[0037] Figure 2 yes Figure 1 A schematic diagram of the cross-task attention mechanism in [the context of the text]. Detailed Implementation
[0038] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.
[0039] Example 1
[0040] This embodiment provides a vulnerability assessment method based on the fusion of multimodal features from code and text and cross-task attention, such as... Figure 1 As shown, it includes the following steps:
[0041] Step 1: First, obtain vulnerability data, which includes vulnerable function code and vulnerability description text. Then, preprocess the vulnerable function code and vulnerability description text respectively.
[0042] Specifically, for the vulnerable function code: firstly, regular expressions are used to remove blank lines and comments from the code to ensure that the code processed during analysis is more concise and accurate. Simultaneously, each line of statement is numbered. In this embodiment, a tree-sitter is used to parse it into an AST (Abstract Syntax Tree). Then, a syntax-based control flow graph (CFG) is constructed based on the AST. During the construction, only statement nodes in the AST are considered, ignoring some finer-grained nodes, resulting in higher construction efficiency.
[0043] Next, several representative execution paths are traversed from the CFG graph, and the function is represented using the execution path from the CFG's entry node to the exit node. This alleviates the problem of large code size, which can lead to a very large graph (like cfg, cpg, etc.) and prevent the model from learning effectively.
[0044] In this embodiment, the main idea behind filtering execution paths is to ensure that the paths are as diverse and short as possible, so as to effectively and completely represent the function. Assuming we need to filter m paths, for the first m-1 paths, we select short and distinct paths; while for the last path, the focus is on maximizing the coverage of more code nodes.
[0045] Furthermore, regarding vulnerability description text: it's understandable that since vulnerability description text contains a lot of useless information, it needs to be filtered out. Vulnerability elements are key information points in the vulnerability description text (such as Impact, Attacker, Cause, etc.). Therefore, using vulnerability elements can remove a lot of irrelevant information from the vulnerability description text.
[0046] Specifically, in this embodiment, the vulnerability description text is simplified by performing word segmentation, removal of stop verbs, and lemmatization, because lemmatization can reduce the vocabulary size, while word segmentation at the word granularity can maximize the integrity of semantic information. Then, CT-FVA uses the pre-trained named entity recognition model BERT-MRC to extract some key vulnerability elements describing the vulnerability from the vulnerability description text.
[0047] Step 2: Extract features from the preprocessed execution path and vulnerability elements, and use a cross-task attention mechanism to fuse multi-task related features.
[0048] Specifically, for the multiple execution paths extracted from the vulnerable code function, it is necessary to consider the characteristics of each path and the relationships between them to represent the function code. For each path, the nodes (statements) in the path are arranged in code order, and then tokenized and embedded using Codebert to obtain a path vector. Multiple path vectors are concatenated to form a path matrix. Next, for each evaluation task, seven independent CNN convolutional neural networks are used, employing 3x3 convolutional kernels to convolve with the path matrix to learn the relationships between different paths, obtaining the initial code features for each evaluation task.
[0049] The vulnerability elements extracted from the vulnerability description text are connected using commas. Glove is then used for initial word embedding, which trains a dense vector for each word by statistically analyzing the co-occurrence frequency of words throughout the text, ensuring that semantically similar words are close in vector space. Finally, for each evaluation task, seven independent CNN convolutional neural networks are used to further extract the initial text features for each evaluation task.
[0050] Next, a cross-task attention mechanism is used on the initial code features and initial text features of each evaluation task to capture the correlation between different evaluation tasks and obtain code fusion features and text fusion features.
[0051] Specifically, such as Figure 2 As shown, cross-task attention mechanisms mainly include two attention mechanisms:
[0052] (1) The relevance-guided attention mechanism aims to use cross-questioning spatial relevance to guide the extraction of features that contribute secondary tasks to primary tasks.
[0053] (2) The self-attention mechanism in secondary tasks aims to discover complementary features that are beneficial to the primary task.
[0054] The same operations are performed at both the code and text levels: For any two evaluation tasks, their initial features are input into the above module to obtain contributing and complementary features. Concatenating these two features yields the interaction features between the two evaluation tasks. The interaction features of each evaluation task are then sequentially concatenated with those of the other evaluation tasks, passed through a 1x1 convolutional layer, a normalization layer, and a ReLU activation function, and finally added to the initial features of that evaluation task to obtain the fused features of that evaluation task.
[0055] Step 3: Based on the extracted code fusion features and text fusion features, the predicted values are obtained through classifiers, and then weighted and fused. Finally, the probability distribution of different labels for the assessment task is obtained through an activation function, and the vulnerability assessment results are output.
[0056] For each evaluation task, the final code and text representations are processed independently using a multilayer perceptron (MLP) to obtain their respective logits. These are then combined by weighted summation to obtain the final evaluation metrics: access vector, access complexity, authentication, confidentiality impact, integrity impact, availability impact, and severity.
[0057] pred i =argmax(softmax(w test ·logits test,i +w code·logits code,i ))
[0058] For the i-th evaluation task, logits test,i ,logits code,i These are the logits represented by text, the logits represented by code, and the output of the last layer of the MLP. test w code These are the weights for text and code, respectively. perd i That is, the predicted label for the i-th evaluation task.
[0059] Subsequently, based on the obtained evaluation metrics, vulnerabilities that need to be prioritized for fixing can be selected and handed over to developers for fixing.
[0060] Example 2
[0061] The difference between this embodiment and Embodiment 1 is that it uses two vulnerability assessment datasets, Bigvul and Primevul, for comparison with five baselines that only use code functions (DeepCVA, FuncRF, FuncLGBM, CodeBERT, FVA) and two baselines that only use vulnerability description text (Le's, TextRNN). All models are trained on the training set, with the training process supervised using the validation set. Models are saved based on their optimal F1-score, and the test dataset is used for testing. This embodiment considers five commonly used performance metrics: F1, MCC, Accuracy, Precision, and Recall. FuncRF, FuncLGBM, Le's, and TextRNN treat each CVSS metric as an independent classification task, thus requiring multiple models, while the others involve multi-task learning and therefore use only one model.
[0062] Tables 1 and 2 show the average values of accuracy, precision, recall, f1, and mcc for seven evaluation tasks on the BigVul and PrimeVul datasets, respectively, for different vulnerability assessment methods:
[0063] Table 1
[0064]
[0065]
[0066] Table 2
[0067]
[0068] As shown in Tables 1 and 2, on the BigVul dataset, among the source code-based methods, FVA achieves the best performance in all metrics except for Precision. Among text-based methods, TextRNN outperforms the Le method. The method provided in this embodiment achieves the best performance in all four performance metrics except for Recall (0.780): Accuracy (0.903), Precision (0.895), F1-score (0.800), and MCC (0.758). In the F1 score, the method in this embodiment improves performance by 0.6% and 6.8% compared to FVA and TextRNN, respectively, while in the MCC score, it improves by 4.3% and 18.3%, respectively. On the PrimeVul dataset, among the source code-based methods, CodeBERT achieves the best overall performance. Among text-based methods, TextRNN performs best. The method in this embodiment achieves the best performance across five metrics: Accuracy (0.816), Precision (0.703), Recalculation (0.653), F1-score (0.671), and MCC (0.584). In the F1 score, this method improves upon CodeBERT and TextRNN by 15.1% and 14.1%, respectively, while in the MCC score, it improves by 45.6% and 37.4%, respectively.
[0069] Tables 3 and 4 show the results of three variations of this method on the BigVul and PrimeVul datasets, respectively. The first variation uses only the vulnerability code, directly inputting the features from the code execution path passed through a CNN into the MLP without using a cross-task attention mechanism; the CNN is shared. The second variation uses only the vulnerability description text, directly inputting the extracted vulnerability elements passed through a CNN into the MLP without using a cross-task attention mechanism. The third variation combines the first two variations, weighting and summing their predicted logits to obtain the final result.
[0070] Table 3
[0071]
[0072] Table 4
[0073]
[0074] Tables 3 and 4 show the performance of the three variants: Element, Path, and Path+Element. On the BigVul dataset, the Path+Element variant improves performance in Acc, Precision, F1, and Mcc metrics by 0.9%–6.0%, 7.6%–23.5%, 0.3%–5.8%, and 5.0%–19.2%, respectively, only slightly lower than the Element variant in Recall. On the PrimeVul dataset, the Path+Element variant improves performance in Acc, Precision, Recall, F1, and Mcc metrics by 1.6%–9.1%, 2.1%–21.6%, 3.9%–4.3%, 4.6%–8.2%, and 6.7%–34.7%, respectively. This verifies the effectiveness of combining vulnerability code and vulnerability description text.
[0075] Furthermore, the effectiveness of this design component can be seen by comparing the Path+Element variant in the table with the method of this embodiment. On the BigVul dataset, the method of this embodiment improves the Acc, Recall, F1, and Mcc metrics by 0.7%, 2.5%, 2.0%, and 1.1%, respectively, only slightly lower than the Path+Element variant in Precision. On the PrimeVul dataset, the method of this embodiment improves the Acc, Precision, Recall, F1, and Mcc metrics by 4.3%, 5.4%, 13.0%, 12.9%, and 14.1%, respectively. This validates the effectiveness of the cross-task attention mechanism component and facilitates the evaluation of the overall performance of the task.
[0076] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.
Claims
1. A vulnerability assessment method based on code and text multimodal feature fusion and cross-task attention, characterized in that, The method comprises the following steps: Step 1, obtaining and preprocessing vulnerability data, wherein the vulnerability data comprises vulnerability function code and vulnerability description text; The preprocessing of the vulnerability function code comprises: converting the code into a control flow graph, and generating multiple execution paths from the control flow graph; The preprocessing of the vulnerability description text comprises: extracting vulnerability elements from the vulnerability description text; Step 2, respectively extracting features from the preprocessed execution paths and vulnerability elements, and fusing the multi-task correlation features by using a cross-task attention mechanism to obtain code fusion features and text fusion features; Step 3, based on the code fusion features and the text fusion features, respectively obtaining prediction values by using a classifier, and performing weighted fusion to output a vulnerability evaluation result.
2. The vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention according to claim 1, characterized in that, The specific steps of converting the code into a control flow graph comprise: using a regular expression to remove blank lines and comments in the code, and marking line numbers for each line of statements; using a syntax parsing tool to parse the code into an abstract syntax tree; constructing a control flow graph based on the abstract syntax tree.
3. The vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention according to claim 2, characterized in that, When constructing the control flow graph, only statement nodes are retained.
4. The vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention according to claim 1, characterized in that, The multiple execution paths generated from the control flow graph comprise: traversing paths from an entry node to an exit node; selecting multiple paths, wherein the first m-1 paths are short and different paths, and the mth path covers code nodes.
5. The vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention according to claim 1, characterized in that, The method for extracting vulnerability elements comprises: performing word segmentation, stop word removal and lemmatization operations on the text; using a pre-trained BERT-MRC model to extract vulnerability elements.
6. The vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention according to claim 1, characterized in that, In step 2: the feature extraction of the code execution path comprises: using a CodeBERT model for encoding, then extracting features by using multiple independent convolutional neural networks, and finally fusing multi-task features by using a cross-task attention mechanism to obtain multiple code fusion features; the feature extraction of the text vulnerability element comprises: using a GloVe word embedding model for initial embedding, then extracting features by using multiple independent convolutional neural networks, and finally fusing multi-task features by using a cross-task attention mechanism to obtain multiple text fusion features.
7. The vulnerability assessment method based on code and text multimodal feature fusion and cross-task attention according to claim 6, characterized in that, The cross-task attention mechanism comprises a relevance-guided attention mechanism and a self-attention mechanism.
8. The vulnerability assessment method based on code and text multimodal feature fusion and cross-task attention according to claim 7, characterized in that, In step 2, the relevance-guided attention mechanism uses cross-query space relevance to guide the extraction of the contribution features of the secondary task to the primary task; and the self-attention mechanism is used to discover complementary features of the primary task.
9. The vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention according to claim 1, characterized in that, In step 3: for each evaluation task, the code fusion features and the text fusion features thereof are respectively processed by using a multilayer perceptron classifier to obtain respective logits; for each evaluation task, the logits of the code and the text are weighted and summed, and then an activation function is used to obtain a probability distribution of different labels of the evaluation task, and finally the CVSS evaluation index is determined.
10. The vulnerability assessment method based on code and text multimodal feature fusion and cross-task attention according to claim 9, characterized in that, The CVSS evaluation index comprises: access vector, access complexity, authentication, confidentiality impact, integrity impact, availability impact and severity.
11. The vulnerability assessment method based on code and text multi-modal feature fusion and cross-task attention according to claim 1, characterized in that, The method is used for function-level vulnerability severity evaluation, and the vulnerabilities to be repaired are preferentially screened according to the evaluation result.