A cross-programming language vulnerability detection method based on image-text attention alignment

Abstract

The present application relates to the technical field of computer software security and deep learning cross-language source code analysis, and particularly relates to a cross-programming language vulnerability detection method based on graphic-text attention alignment. In view of the poor cross-language generalization ability of the existing code vulnerability detection model, the present application proposes a multi-view feature alignment cross-language vulnerability detection scheme. The scheme comprises: using a syntax parsing engine to construct heterogeneous language source code into a single code property graph (CPG); using a pre-trained code large model and a graph neural network to extract the semantic and CPG topological structure features of the code in parallel; using a cross-attention mechanism to realize lossless alignment and fusion of multi-view features; and jointly using a supervised contrast learning loss and a multi-classification cross-entropy loss to train the model. Through the mechanism, in a unified high-dimensional feature space, the representation distance of similar defect samples under cross-language is effectively reduced, and thus the cross-language vulnerability migration detection precision and generalization performance of the model are significantly enhanced.

Description

Technical Field

[0001] This invention relates to the fields of computer software security and deep learning source code analysis technology, and in particular to a cross-programming language vulnerability detection method based on graph-text attention alignment. Background Technology

[0002] With the rapid expansion of software scale and the widespread use of open-source components, source code vulnerabilities have become one of the biggest threats to cybersecurity. In actual business operations, different projects often use different programming languages ​​(such as C, Java, Python, Rust, etc.) for mixed development. The syntactic representation of vulnerability logic varies greatly in different programming languages, but their underlying malicious intent (such as CWE-79 cross-site scripting, CWE-89 SQL injection, etc.) often originates from the same source.

[0003] The existing technology has the following main problems:

[0004] 1. Poor cross-language generalization ability: Traditional rule-based or unimodal deep learning models (such as those using only plain text representations) heavily rely on specific syntactic features of the training set. After being trained on source languages ​​such as C / C++, these models fail to extract features when faced with unseen target languages ​​(such as Ruby or Rust), making cross-language transfer difficult.

[0005] 2. Fragmented multi-view structure disrupts underlying logic: Existing graph neural network detection schemes typically treat the Abstract Syntax Tree (AST), Control Flow Graph (CFG), and Data Flow Graph (DFG) as three independent graph structures, which not only leads to huge memory overhead but also breaks the complete chain of evidence for vulnerabilities, namely "syntax definition - control jump - data pollution".

[0006] 3. Graph pooling suffers from catastrophic noise dilution: When aligning code lengths, a large amount of padding is typically introduced. Traditional global mean pooling includes these invalid zero nodes in its calculations, causing genuine weak vulnerability features to be severely diluted by massive noise.

[0007] 4. Multimodal fusion is prone to gradient oscillation: When a pre-trained large language model is directly connected to a randomly initialized graph neural network, the inconsistency in feature distribution can easily lead to gradient explosion or model collapse.

[0008] 5. Lack of adaptive decision-making under extremely imbalanced data: The proportion of vulnerability samples in real-world scenarios is extremely low (e.g., 15:1). Traditional models use a fixed decision threshold of 0.5, resulting in an extremely high false positive rate, rendering static security audit tools unusable.

[0009] Therefore, there is an urgent need for a deep learning vulnerability detection solution that can eliminate grammatical differences between heterogeneous languages, maintain unified structural logic, remove alignment noise, and has cross-language generalization capabilities. Summary of the Invention

[0010] The purpose of this invention is to provide a cross-programming language vulnerability detection method based on graph-text attention alignment, so as to solve the technical problems of poor cross-language generalization of long text code, dilution of graph features, and difficulty in multi-view alignment.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] This invention no longer treats heterogeneous source code as isolated character sequences, but instead constructs a complete closed loop of "CPG graph construction—image and text dual-stream extraction—lossless mask pooling—joint loss cross-language alignment—dynamic decision-making" through a unified parsing engine, dual-stream parallel encoding, attention mask absolute alignment, and supervised contrastive learning.

[0013] 1. The data flow graph conversion module is responsible for eliminating the literal differences between different programming languages ​​and converting heterogeneous source code into a unified token sequence containing words and a single code attribute graph (CPG).

[0014] 2. The dual-stream feature joint encoding module is responsible for extracting code semantic vectors using a cross-language pre-trained code model and extracting deep logical features of the CPG topology dimension using a graph convolutional network (GCN).

[0015] 3. The attention mask lossless pooling module is responsible for introducing the Transformer attention mask as a graph node gate to accurately remove padding noise and achieve 1:1 absolute alignment and layer normalization of the graph and text feature space;

[0016] 4. The feature splicing and perception module is responsible for splicing multimodal features and generating binary classification prediction logic values ​​(safe or vulnerable) to indicate the security status of the code through multilayer perceptron mapping.

[0017] 5. The cross-language dynamic decision optimization module is responsible for adaptively determining the optimal classification confidence threshold on the cross-language validation set based on the precision-recall (PR) curve under extremely imbalanced positive and negative sample distributions, thereby achieving high-precision binary classification recognition of cross-language vulnerability features in a unified feature space.

[0018] Compared with the prior art, the present invention has at least the following beneficial effects:

[0019] 1. Achieve high-level cross-language generalization: By aligning the feature space with supervised contrastive loss (SupConLoss), the model can still detect unseen heterogeneous target programming language vulnerabilities with high accuracy even when trained using only a portion of the language.

[0020] 2. Eliminate feature fragmentation and memory bottlenecks: By constructing a single fused CPG topology graph to replace the isolated AST / CFG / DFG, the context logic chain of vulnerability triggering is fully preserved, and the computational complexity of graph neural networks is significantly reduced.

[0021] 3. Solving the feature dilution problem: Introducing the Attention-Masked Pooling mechanism to accurately mask placeholder noise and significantly improve the model's ability to capture weak vulnerability features.

[0022] 4. Improve accuracy under extremely imbalanced data: Through adaptive PR curve optimization and gradient anti-explosion strategy, the false alarm rate in industrial applications is effectively reduced, which has extremely high practical deployment value. Attached Figure Description

[0023] Figure 1 This is a workflow framework diagram of a cross-programming language vulnerability detection method based on graph and text attention alignment according to the present invention. It is used to show the complete execution process from code data preprocessing, dual-stream feature extraction, pooling alignment to dynamic decision boundary optimization.

[0024] Figure 2 This is a schematic diagram of the overall system (network) structure of the present invention, used to illustrate the logical connections and feature transfer relationships between the pre-trained code large model, multi-layer graph convolutional network, feature splicing layer and multi-layer perceptron (MLP) classifier.

[0025] Figure 3 This is a schematic diagram of the attention mask lossless pooling processing mechanism of the present invention, used to illustrate the core physical alignment process of mask matrix expansion, element-wise physical multiplication, effective node summation and layer normalization. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Unless otherwise defined, the technical or scientific terms used herein should have the ordinary meaning understood by those skilled in the art to which this invention pertains. The terms "comprising" and similar expressions used herein mean that the element or object preceding the word covers the element or object listed following the word and its equivalents, but does not exclude other elements or objects.

[0027] This invention provides a cross-programming language vulnerability detection method based on graph-text attention alignment, comprising the following steps:

[0028] S1.1 Establish a cross-language programming vulnerability binary classification detection workflow, which includes: a data stream graph conversion module, a dual-stream feature joint encoding module, an attention mask lossless pooling module, a feature splicing perception module, and a dynamic decision boundary optimization module;

[0029] S1.2 The data flow graph conversion module performs code preprocessing, flattens the source code into a token sequence, and simultaneously parses the syntax tree and data / control flow of the code to generate a single code attribute graph (CPG) edge index, and generates an attention mask (attention_mask) with a corresponding effective sequence length.

[0030] S1.3 The dual-stream feature joint encoding module performs parallel extraction of multimodal features, inputs the token sequence into the pre-trained code large model to extract global text semantic vectors, and simultaneously uses a multi-layer graph convolutional network (GCN) to aggregate features along the CPG edges to extract deep structured node tensors;

[0031] S1.4 The attention mask is extracted by the attention mask lossless pooling module as the graph node gating signal, accurately removing the placeholder (padding) node noise introduced by length alignment, calculating the weighted mean of the effective node features and performing layer normalization to align the feature space.

[0032] S1.5 The feature splicing perception module directly splices (Concat) the text semantic vector with the normalized graph modal features and inputs it into the multilayer perceptron (MLP) to generate a binary classification logic judgment value used to indicate the probability of vulnerabilities.

[0033] S1.6 During the model training phase, calculate the binary cross-entropy loss, suppress gradient oscillations through gradient backpropagation and gradient pruning strategies, and continuously update the bimodal network parameters.

[0034] S1.7 During the model evaluation phase, the dynamic decision boundary optimization module adaptively fits the optimal classification confidence threshold on the validation set based on the precision-recall (PR) curve, and outputs the final vulnerability detection warning result based on this threshold under extremely imbalanced data distribution.

[0035] In step S1.1, a workflow architecture for a cross-programming language vulnerability detection method based on graph-text attention alignment is designed. This workflow architecture consists of five core parts: a data flow graph transformation module, a dual-stream feature joint encoding module, an attention mask lossless pooling module, a feature splicing perception module, and a dynamic decision boundary optimization module.

[0036] The workflow framework diagram of a cross-programming language vulnerability detection method based on graph-text attention alignment provided by this invention is as follows: Figure 1 As shown.

[0037] The overall system architecture diagram of the cross-programming language vulnerability detection method based on graph-text attention alignment provided by this invention is as follows: Figure 2 As shown.

[0038] The processing principle diagram of the attention mask lossless pooling mechanism in the cross-programming language vulnerability detection method based on graph-text attention alignment provided by this invention is shown below. Figure 3 As shown.

[0039] 1) Data Flow Graph Conversion Module: This module aims to receive, parse, and standardize the raw tensor files in the vulnerability dataset, providing a unified graph-text alignment input for subsequent multimodal networks. Through customized data collating operators (such as `custom_graph_collate`), this module achieves unified distribution of flattened feature matrices (x), control and data flow edge indices (edge_index), attention masks, and true vulnerability labels, thereby avoiding feature misalignment issues caused by truncation and padding of long text code.

[0040] 2) Dual-stream Feature Joint Encoding Module: This module aims to extract lexical semantic features and global structural features of the input code in parallel. By instantiating a pre-trained large code model (such as Microsoft / GraphCodeBERT-Base) and a multi-layer graph convolutional network (GCNConv), this module achieves joint capture of local character dependency analysis and cross-statement logical links (such as control flow jumps and variable passing), thereby enabling the model to possess a deep code intent understanding capability that surpasses simple text matching.

[0041] 3) Attention Mask Lossless Pooling Module: This module aims to solve the catastrophic noise dilution problem caused by traditional graph neural networks when processing padding-length codes. Through core mechanisms such as mask dimension expansion, matrix element-wise multiplication, effective node tensor summation, and distribution layer normalization (LayerNorm), this module achieves a 1:1 absolute physical alignment between the graph topological space and the text semantic space, thereby significantly improving the model's ability to represent weak vulnerability features.

[0042] 4) Feature Concatenation and Awareness Module: This module aims to perform high-dimensional decision fusion on the aligned bimodal features and output binary classification prediction probabilities conforming to a predefined format. This module achieves a smooth mapping of the security and vulnerability binary classification boundary through mechanisms such as feature tensor concatenation (Concat), linear dimensionality reduction, nonlinear activation (GELU), and anti-overfit deactivation (Dropout), and outputs logical judgment values ​​(Logits) indicating the current code vulnerability risk.

[0043] 5) Dynamic Decision Boundary Optimization Module: This module aims to address the extremely imbalanced positive and negative sample distribution problem in real-world code security audits (e.g., a secure code to vulnerable code ratio of 15:1). This module employs mechanisms such as extracting global validation set prediction probabilities, fitting precision-recall (PR) curves, calculating the global harmonic mean (F1-Score), and adaptive threshold determination. It abandons the traditional 0.5 static decision threshold used in industry, thereby significantly suppressing the false positive rate of static code scanning while maintaining recall.

[0044] During step S1.2, the data stream graph conversion module acquires the preprocessed tensor data and performs padding and alignment. The preset maximum sequence length is set to... Batch size is For lengths that are insufficient The code token sequence is padded with invalid placeholders at the end, and a dimension of [dimensional value] is generated. attention mask (Where the position corresponding to the actual token is 1, and the placeholder position is 0). The abstract syntax tree (AST) and data flow graph of the extracted code are merged to form a single code attribute graph (CPG) edge index matrix. .

[0045] During step S1.3, the dual-stream feature joint encoding module performs parallel multimodal feature extraction. For the text stream, the sequence tensor is fed into the large language base, and the extracted shape is... Hidden layer state tensor And extract the first [CLS] vector; for the graph stream, After flattening into a two-dimensional matrix, along with The input is a two-layer graph convolutional network. The first layer, GCNConv, captures local syntactic dependencies and performs ReLU activation. The second layer, GCNConv, captures cross-statement pollution links, and the final output has the same shape. Structured feature tensor .

[0046] During step S1.4, the attention mask lossless pooling module performs... Perform precise noise reduction. In a preferred embodiment, Add a dimension (unsqueeze(-1)) to the end and convert to floating-point, resulting in The mathematical expression for the pooling process is: [The following is a continuation of the previous sentence, likely due to an error in the original text.] and Perform element-wise multiplication to mask the eigenvalues ​​of all filling nodes; then sum along the sequence length dimension and divide by the non-zero count value of the effective mask: The 1e-9 increment in the denominator is a local constant designed to prevent errors caused by division by zero. After pooling, the clean output image is locally represented by the LayerNorm function to eliminate gradient oscillations before cross-modal fusion.

[0047] In step S1.5, the feature splicing perception module performs high-dimensional decision fusion on the dual-modal features. Specifically, the 768-dimensional text [CLS] semantic vector and the normalized 768-dimensional graph modality vector are directly spliced ​​in the feature dimension to form a 1536-dimensional fusion feature matrix. This matrix is ​​then passed through a dimension reduction linear layer, a non-linear activation function (GELU), and a random deactivation layer (Dropout) with a deactivation rate set to 0.1. Finally, it is mapped through the output linear layer to generate a feature vector of length 2, which corresponds to the logical judgment value of predicting the current code as "safe (0)" or "vulnerable (1)".

[0048] In step S1.6, the network prediction error is calculated using the multi-class cross-entropy loss function during the model training phase, and the AdamW optimizer is used to perform gradient updates. In this embodiment, all multimodal network modules apply a globally uniform baseline learning rate (e.g., 2e-5), and a maximum gradient norm pruning strategy (max_norm=1.0) is forcibly introduced to ensure the absolute stability of the model during backpropagation of deep image and text features.

[0049] In step S1.7, the dynamic decision boundary optimization module performs adaptive threshold optimization, abandoning the traditional 0.5 static decision threshold used in industry. During the validation phase, the binary classification prediction probabilities and true labels of all samples are extracted. By scanning all potential probability cutoff points, the corresponding precision and recall are calculated, and the harmonic mean F1 score is calculated accordingly. The system will automatically lock the probability index corresponding to the F1 score reaching the global maximum value, extract this threshold (e.g., 0.1375) as the optimal classification decision boundary for this batch of imbalanced datasets, and use this boundary to perform the identification of vulnerabilities in the final test set.

[0050] In a preferred embodiment of vulnerability detection, a C language function source code containing a "buffer overflow (CWE-119)" vulnerability is input. The system first converts the code into a 512-byte aligned token sequence using a preprocessing module and parses the CPG topology graph structure. Then, a dual-stream encoding module simultaneously extracts the code's lexical semantics and structural topology features. In the core pooling stage, attention mask gating precisely eliminates redundant graph node interference caused by the 200 padding whitespace characters added at the end due to the short code length. After concatenating the clean features, the classifier outputs a confidence probability of 0.24 that the code contains the vulnerability. Since the system has adaptively scanned and determined the optimal defense threshold for 15:1 extremely imbalanced data to be 0.1375 in previous stages, and 0.24 is greater than 0.1375, the system successfully identifies the vulnerability and issues a vulnerability alert. This mechanism avoids the false negatives that would occur if a traditional static threshold of 0.5 were used.

[0051] Through the above implementation methods, this invention not only overcomes the serious feature dilution and gradient oscillation problems in multimodal fusion, but also utilizes attention mask physical gating and dynamic PR curve optimization mechanisms to achieve a multi-perspective vulnerability detection paradigm with high accuracy, low false positives, and high compliance with industrial-grade code security auditing requirements.

[0052] While embodiments of the present invention have been described in detail above, it will be apparent to those skilled in the art that various modifications and variations can be made to these embodiments. However, it should be understood that such modifications and variations fall within the scope and spirit of the invention as set forth in the claims. Furthermore, the invention described herein may have other embodiments and can be implemented or carried out in various ways.

Claims

1. A cross-programming language vulnerability detection method based on graph-text attention alignment, characterized in that, include: The module includes a data stream graph transformation module, a dual-stream feature joint encoding module, an attention mask lossless pooling module, a feature splicing perception module, and a dynamic decision boundary optimization module. The data flow graph conversion module performs preprocessing and graph construction of heterogeneous source code, eliminates literal differences between different programming languages, flattens the heterogeneous source code into a fixed-length token sequence, and simultaneously parses the syntax and logical edges of the source code, merging them into a single code attribute graph (CPG), outputting a set of graph attribute objects containing graph node edge indices (edge_index), and generating an attention mask (attention_mask) to identify the actual sequence length. The dual-stream feature joint encoding module performs parallel extraction of multimodal features. On the one hand, it inputs the token sequence into a pre-trained code model to extract the first-order classification vector ([CLS]) and the sequence hidden state (last_hidden_state) containing deep contextual semantics. On the other hand, it uses a two-layer graph convolutional network (GCNConv) to perform second-order neighbor information aggregation on the topological space for the node edge indexes in the graph attribute object set to extract the deep structured node tensor of the code logic dimension. The attention mask lossless pooling module performs modal alignment and noise reduction, extracts the attention mask generated by the large code model as a graph node gating signal, shields redundant placeholder node noise introduced by multilingual code length alignment padding operation, accurately calculates the weighted mean graph representation of effective code nodes, and aligns the cross-language feature distribution through layer normalization (LayerNorm) to output the final dimension graph modal feature vector. The feature concatenation perception module performs cross-language decision feature fusion, directly concatenating the first-order classification vector of the text with the graph modality feature vector in the feature dimension, inputting it into a multilayer perceptron containing an activation function and a dropout mechanism, and outputting logical judgment values ​​(Logits) to indicate the probability of vulnerabilities in code of different programming languages. The dynamic decision boundary optimization module performs adaptive threshold optimization. In the scenario of unbalanced positive and negative sample distribution, it constructs a precision-recall (PR) curve based on the predicted probability and real labels collected in the cross-language verification stage, adaptively calculates the F1 score under different thresholds, and determines the optimal classification confidence threshold that maximizes detection efficiency, which serves as the final decision boundary for vulnerability reporting after eliminating language differences.

2. The method according to claim 1, characterized in that, The data input format processed by the data flow graph conversion module is a tensor file with the .pt extension, which contains the flattened node features, edge index structure and binary long integer labels corresponding to whether there are vulnerabilities after parsing the source file.

3. The method according to claim 1, characterized in that, In the dual-stream feature joint encoding module, the pre-trained large code model adopts microsoft / graphcodebert-base, and the output text first classification vector and the hidden layer dimension (hidden_size) of the graph features are both uniformly set to 768 dimensions.

4. The method according to claim 1, characterized in that, In the dual-stream feature joint encoding module, the output of the first-layer graph convolutional network (GCNConv) is filtered by the ReLU activation function and then fed into the second-layer graph convolutional network to capture long-range logical violations or data out-of-bounds risks of cross-language grammatical differences in a unified code attribute graph topology.

5. The method according to claim 1, characterized in that, The noise reduction and averaging logic of the attention mask lossless pooling module is specifically manifested as follows: the extracted attention mask is given a dimension at the end and converted to floating-point type (mask_expanded), multiplied element-wise with the tensor output by the graph convolutional network, and then summed along the sequence dimension. Subsequently, it is divided by the effective count value of the attention mask summed along the sequence dimension. A truncation constraint of not less than 1e-9 is introduced for the effective count value to prevent division by zero anomalies, thereby projecting the code features of different languages ​​to the same alignment space losslessly.

6. The method according to claim 1, characterized in that, In the feature splicing perception module, the multilayer perceptron classifier is composed of a linear mapping layer, a GELU activation function, a Dropout layer with a deactivation rate of 0.1, and a final classification layer connected in series, and the final output is a feature vector containing two categories.

7. The method according to claim 1, characterized in that, During the training phase, the Cross-Entropy Loss function is used, the global learning rate is applied to the parameters of each module, a Cosine Annealing Learning Rate Scheduler is introduced, and the gradient explosion problem during cross-language backpropagation of multi-source feature streams is suppressed by limiting the maximum gradient norm (max_norm=1.0).

8. The method according to claim 1, characterized in that, During the model evaluation phase, the dynamic decision boundary optimization module calculates the precision and recall rates for all candidate thresholds by executing precision_recall_curve, calculates the harmonic mean F1, and takes the threshold corresponding to the maximum value index as the optimal classification confidence threshold. This suppresses the inductive bias of the model in predicting safe samples in an imbalanced detection scenario where the number of vulnerable code is much smaller than the number of safe code.

9. A cross-programming language vulnerability detection method based on graph-text attention alignment, characterized in that, The method includes the following steps: S1.1 The data flow graph conversion module obtains cross-language source code input and performs parsing and preprocessing, extracting the token sequence used to represent semantics, the edge structure index used to represent the code attribute graph (CPG), and the corresponding attention mask. S1.2 The dual-stream feature joint encoding module performs parallel extraction of multimodal features, uses the code big model to extract the global semantic tensor based on the token sequence, and uses the graph neural network to extract the graph node feature matrix based on the edge structure index. S1.3 The attention mask lossless pooling module combines the attention mask to perform pooling noise reduction and masking on the graph node feature matrix, and outputs local mean feature representation to achieve spatial alignment with the global semantic tensor. S1.4 The feature splicing perception module splices the global semantic tensor and the local mean feature representation in dimensions, and inputs the spliced ​​high-dimensional features into the classification network, outputting a logical judgment value used to indicate the probability of code vulnerabilities. S1.5 The dynamic decision boundary optimization module adaptively calculates candidate thresholds based on the predicted logical judgment values ​​and true labels during the model evaluation phase, extracts the optimal classification confidence threshold that maximizes the harmonic mean of precision and recall (F1 score), and performs vulnerability determination on the target language based on this threshold.

Images