Cross-modal semantic bridging method for code graph and large language model

Abstract

The invention discloses a cross-modal semantic bridging method for a code graph and a large language model. The cross-modal semantic bridging method comprises the following steps: converting an original code text into a code attribute graph; inputting the code attribute graph into a code graph encoder of the trained bridging system to obtain a code graph vector; inputting the original code text and the code graph vector into a bridging module of the trained bridging system to obtain a text-graph embedded vector; projecting the text-graph embedded vector into a soft hint vector through a large model projection layer; embedding and splicing tokens of the soft prompt vector, the original code text and the natural language instruction to obtain an input vector; and taking the input vector as a pre-trained LLM to obtain an answer corresponding to the natural language instruction. The invention provides an efficient adaptation method, and effectively solves the problems of structural information loss, shallow logic understanding, poor robustness and high adaptation cost caused by the fact that LLM depends on a linear text sequence in the prior art.

Description

Technical Field

[0001] This invention relates to a fusion technology of code graph representation learning and large language models, and more particularly to a cross-modal semantic bridging method of code graphs and large language models. Background Technology

[0002] Existing LLMs suffer from a fundamental flaw in their code processing: they typically "linearize" complex source code into a flat sequence of tokens. Source code itself contains rich structural semantic information, such as Abstract Syntax Trees (ASTs), Control Flow Graphs (CFGs), and Data Flow Graphs (DFGs). Treating code as plain text, ignoring this inherent structural information, limits LLMs' ability to capture deep semantics and understand complex program logic. This deficiency makes the model highly sensitive to minor perturbations on the code surface (such as variable renaming), leading to completely erroneous code interpretations when variable names are obfuscated.

[0003] To address this problem, existing technologies have primarily explored two approaches: First, there's graph-augmented prompting / serialization. This approach attempts to serialize the graph structure of the code (such as an AST) into a text sequence and then directly input it as context information into the LLM's prompt. However, this method has significant drawbacks. Information dilution and length limitations: Linearizing complex graph structures into text significantly increases the prompt's length, exceeding the context window limitations of existing LLMs. Poor structure preservation: Linear text struggles to effectively preserve complex topological information, leading to dilution of structural knowledge. Experiments also show that this method has limited effectiveness and can sometimes even degrade performance due to the introduction of noise.

[0004] Second, Structure-aware pretraining: Other methods, such as GraphCodeBERT and UniXCoder, attempt to incorporate graph structure information from the code during the model pretraining stage. However, these methods also have limitations: Architectural incompatibility: They are typically designed for small-scale models and rely on fine-tuning for specific downstream tasks. They often require modifications to the model architecture (e.g., adding specialized decoders) for generation tasks, making them incompatible with current mainstream, large-scale, end-to-end instruction-following LLM architectures. High computational cost: Extending this structure-aware pretraining and fine-tuning paradigm to large language models with billions or even tens of billions of parameters is extremely computationally expensive and even impractical.

[0005] Furthermore, while existing parameter efficient fine-tuning (PEFT) methods (such as LoRA) are widely used, they are essentially still text sequence-based fine-tuning and fail to address the inherent structural information loss problem in LLMs. Therefore, they perform poorly in understanding deep program logic and resisting code perturbations (such as variable renaming). Consequently, existing technologies lack a method that can efficiently and plug-and-playly inject rich structural semantics of code into existing, pre-trained, frozen large-scale language models without modifying the LLM architecture itself. Summary of the Invention

[0006] To address the aforementioned shortcomings of existing technologies, this invention provides a cross-modal semantic bridging method for code graphs and large language models, which solves the problems of existing technologies ignoring the inherent structural information of code, having low code comprehension capabilities, high costs, and poor robustness.

[0007] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is to convert the original code text into a code attribute diagram; The code attribute graph is input into the code graph encoder of the trained bridging system to obtain the code graph vector; The original code text and code graph vector are input into the bridging module of the trained bridging system to obtain text-graph embedding vectors; The text-graph embedding vector is projected into a soft cue vector through a large model projection layer; The input vector is obtained by concatenating the token embeddings of the soft prompt vector, the original code text, and the natural language instructions. The input vector is used as a pre-trained LLM to obtain the answer corresponding to the natural language instruction.

[0008] The beneficial effects of this invention are as follows: 1. This invention achieves complete decoupling between the code structure understanding module (bridging module) and the general language understanding module (LLM). This differs from existing methods such as "structure-aware pre-training" (requires retraining and is incompatible) and "LoRA fine-tuning" (modifies internal parameters and carries the risk of forgetting). By keeping the core parameters of the LLM frozen, this invention trains only an external Bridge module, which greatly reduces adaptation costs (saving over 99%) and allows it to be "plug-and-play" adapted to any existing or future LLM, exhibiting extremely high versatility and cost-effectiveness.

[0009] 2. Existing "GraphText" methods attempt to convert graphs into text, but suffer from fundamental drawbacks such as information loss, inefficiency, and noise interference. This invention innovatively avoids converting graphs into text. Instead, it uses a Bridge module to compress the complex structural semantics of the CPG graph into a set of high-information-density "soft cue" vectors and directly injects them into the embedding space of the LLM. Compared to text serialization, this vector injection method offers higher fidelity, greater information density, and does not occupy the LLM's context window, significantly improving the efficiency of information injection. Attached Figure Description

[0010] Figure 1 The flowchart illustrates a method for cross-modal semantic bridging between code graphs and large language models, provided as an example. Detailed Implementation

[0011] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0012] like Figure 1 As shown, in one embodiment of the present invention, a method for cross-modal semantic bridging of code graphs and large language models includes the following steps: S1. Convert the original code text into a code attribute graph (CPG). The code attribute graph includes syntax tree edges (AST) of the encoding syntax level (such as function definition, containing code block), control flow edges (CFG) of the encoding control flow level (such as conditional statement, correct branch and incorrect branch), and data flow edges of the encoding data dependency level (such as variable a' flowing to variable x').

[0013] S2. Input the code attribute graph into the code graph encoder (CGE) of the trained bridging system to obtain the code graph vector.

[0014] The code graph encoder is a A graph neural network with layers, where the update operation of each layer includes an attention mechanism modulated by edge features; where nodes For neighboring nodes The attention coefficient is:

[0015] In the formula, For the first Layer nodes For neighboring nodes Attention coefficient; , , The first The query weight matrix, key weight matrix, and value weight matrix of the layer; For the first Layer nodes Node characteristics; For the first Layer nodes Node characteristics; As edge features, The dimension of the key; Aggregating nodes using attention mechanisms The node obtained after the neighbor node is represented as:

[0016] In the formula, For the aggregated nodes Node characteristics; , All are learnable weight matrices. For nodes The set of neighbors.

[0017] The training process of the code graph encoder is as follows: Randomly obtain code attribute graphs (CPGs). Specifically, collect large-scale unlabeled source code (e.g., the CodeSearchNet dataset, approximately 260,000 Python samples), and use code parsing tools such as tree-sitter to generate an AST (Abstract Syntax Tree) for each code. Based on the AST, extract CFG (Control Flow Graph) edges (e.g., true_branch, loop_exit) and DFG (Data Flow Graph) edges (e.g., flows_to, contributes_to) through static analysis. Combine the AST, CFG, and DFG into a heterogeneous CPG, and use a pre-trained encoder (e.g., UniXcoder-base) to initialize features for the nodes and edges in this code attribute graph, resulting in randomly obtained code attribute graphs.

[0018] The code attribute graph is randomly augmented twice to obtain the first augmented sample and the second augmented sample. The first and second augmented samples are input into the code graph encoder to obtain the first and second code graph vectors. By reducing the InfoNCE loss value, the distance between the first code graph vector and the second code graph vector is narrowed until the InfoNCE loss value no longer decreases; Randomly sample node pairs from the code attribute graph and input them into the code graph encoder; output the edge type between node pairs. The ability of the code graph encoder to identify edge types is optimized by reducing the cross-entropy loss value until the cross-entropy loss value no longer decreases. The total training loss of the code graph encoder is:

[0019] In the formula, This represents the total training loss value of the code graph encoder. For InfoNCE loss weights, For the InfoNCE loss function, For cross-entropy loss weights, This is the cross-entropy loss function. After this step of training is completed, CGE is frozen.

[0020] S3. Input the original code text and code graph vector into the bridging module of the trained bridging system to obtain the text-graph embedding vector.

[0021] The bridge module is a The transformer consists of layers, each layer comprising a text encoder, a shared self-attention mechanism, a cross-attention mechanism, and a feedforward network connected in sequence.

[0022] The specific method for obtaining the text-graph embedding vector is as follows: The original code text is input into the text encoder in the bridging module to obtain the code text embedding vector; By using a shared self-attention mechanism to exchange information between the code text embedding vector and the query vector, a query vector with fused text information is obtained, the expression of which is:

[0023] In the formula, For the first The query vector after fusing text information in the layer. For the first The code text embedding vector after information exchange within the layers This indicates the self-attention mechanism. For the first The query vector after fusing text information in the layer. For the first The embedded vector of code text after information exchange within the layer. When hour, This refers to the learnable query vectors in the bridging module. This is a code text embedding vector encoded by a text encoder from the original code text.

[0024] Using the code graph vector as key K and value V, and the query vector after fusing text information as query Q, the key, value, and query are input into a cross-attention mechanism to obtain a vector injected with graph structure information, the expression of which is:

[0025] In the formula, For the first Vectors that inject graph structure information into layers. This represents the cross-attention mechanism. Represents a code graph vector.

[0026] The query vector after fusing textual information and the vector injected with graph structure information are input into the feedforward network to obtain the final output of this layer, which is expressed as follows:

[0027] In the formula, , All indicate the bridging module number. The output of the layer.

[0028] go through After the above processing, the final text-graph embedding vector is output. .

[0029] The bridging module is trained based on three complementary loss functions, including: Image-text comparison loss function:

[0030] In the formula, This represents the loss value for image-text comparison. express loss function Represents a text-graph embedding vector. Embed vectors for code text. Learnable temperature; Image-text matching loss function:

[0031] In the formula, For image-text matching loss value, For the sample size, For the first A true value, For the first One predicted value, Graph-guided text generation loss function:

[0032] In the formula, To guide the generation of loss values ​​for the text, For the first The first sample One token, Indicates the first The text-graph embedding vectors corresponding to each sample; The total training loss of the bridging module is:

[0033] In the formula, This represents the total training loss of the bridging module.

[0034] After training, the bridging module is capable of fusing and compressing arbitrary code and its code graph vectors into high-quality embeddings. The ability.

[0035] S4. Project the text-graph embedding vector into a soft cue vector through a large model projection layer.

[0036] S5. Concatenate the token embeddings of the soft prompt vector, the original code text, and the natural language instructions to obtain the input vector.

[0037] S6. Use the input vector as a pre-trained LLM to obtain the answer corresponding to the natural language instruction.

[0038] The pre-trained code graph encoder, the pre-trained bridging module, and the large model projection layer constitute the bridging system; training the code graph encoder and training the bridging module are executed independently of each other.

[0039] The specific training method for the bridging system is as follows: Obtain training samples, including raw code text, natural language instructions (e.g., "summarize code functionality"), and standard answers; After the original code text is converted into a code attribute graph, it is input into the bridging system. The trained code graph encoder receives the code attribute graph and outputs a code graph vector. The bridging module receives the code graph vector and the original code text and outputs a text-graph embedding vector. The text-graph embedding vector is projected into a soft cue vector through a large model projection layer (a simple linear layer). The dimension of the soft cue vector is consistent with the word embedding dimension of the LLM.

[0040] The input vector is obtained by concatenating the token embeddings of the soft suggestion vector, the original code text, and the natural language instructions. , ,in Represents the soft cue vector. Representing natural language instructions. Represents the original code text; The input vector is fed into a pre-trained LLM to obtain the answer corresponding to the natural language instruction; The loss between the calculated answer to the natural language instruction and the standard answer is expressed as follows:

[0041] In the formula, This represents the total training loss value of the bridging system. For all trainable parameters of the bridging system, As the standard answer, Conditional probability distributions generated for large language models For the input vector, The first in the standard answer sequence One token, The first in the standard answer sequence All tokens preceding this token. Calculate the total training loss of the bridging system with respect to The gradient is used to optimize the gradient using gradient descent. The trained bridging system is obtained. During the training of the bridging system, all parameters of the LLM remain frozen and do not receive any gradient updates.

[0042] In this way, the present invention provides an efficient adaptation method where all task-specific, code structure-related knowledge is learned and stored in an external, relatively small-parameter bridging module. This enables the present invention to significantly improve the logical understanding, functional correctness, and robustness of LLM in tasks such as code summarization and code translation without incurring high LLM full-parameter fine-tuning costs and with inference speeds far faster than methods such as LoRA. By injecting the graph structure semantics (CPG) of the code into a frozen large language model (LLM) through an external trainable bridging module, the present invention effectively solves the problems of structural information loss, superficial logical understanding, poor robustness, and high adaptation costs caused by the reliance of LLM on linear text sequences in existing technologies.

[0043] The technical effects achievable by this invention have been confirmed through specific scientific experiments (as shown in Tables 1, 2, 3, and 4).

[0044] Table 1

[0045] Table 2

[0046] Table 3

[0047] Table 4

[0048] Table 1 shows the comparison results of various indicators when using different methods for original code text summarization and translation, where CGBRIDGE represents the method provided by this invention. In the code summarization task: The "LLM-as-a-Judge" (LLM-J) score, which measures semantic and logical consistency, shows that the CGBRIDGE method of this invention (e.g., achieving 3.23 points on the Qwen2.5-Coder-7B model) is significantly higher than the traditional LoRA fine-tuning baseline (2.83 points), demonstrating that the generated summary is logically more consistent and accurate. In the code translation task: "Execution Accuracy" (EA), which measures functional correctness, is the most critical indicator. The method of this invention (e.g., achieving 98.26% EA on Qwen2.5-Coder-7B) outperforms the LoRA baseline (97.17%); the advantage is even more pronounced on Qwen2.5-Coder-1.5B (89.01% vs 84.68%), proving that this invention can generate more functionally correct and executable code.

[0049] Table 2 illustrates the robustness of different methods when renaming variables. When variable names are obfuscated, the performance (LLM-J score) of the text-based LoRA method drops sharply (e.g., -15.2%). The GraphText method, which serializes graphs to text, fails completely (-28.4%). Since this invention relies on structural information in the code graphs (Data Flow Graph (DFG) and Control Flow Graph (CFG)) rather than variable names, its performance is almost unaffected, with the LLM-J score decreasing by only -0.3% to -1.6%. This data strongly demonstrates that this invention successfully shifts LLM from relying on "surface text" to relying on "program structure," solving the fundamental problem of poor robustness in existing LLM technologies.

[0050] Table 3 shows the ablation experiments of different code attribute graph components. The experiments demonstrate that the highest execution accuracy (EA) can only be obtained when the syntax tree (AST), control flow (CFG), and data flow (DFG) are used simultaneously. This proves that the synergistic effect of these three structural information is crucial for achieving optimal functional correctness and confirms the effectiveness of using CPG (fusion of AST, CFG, and DFG) in the technical solution of this invention.

[0051] Table 4 illustrates the efficiency of different methods. Experiments demonstrate that the "plug-and-play" and "frozen LLM" architectures of this invention bring significant efficiency improvements: Technical performance (inference speed): During inference (generation), the present invention is more than 4 times faster than the LoRA method (e.g., on a 1.5B model, the present invention takes approximately 215ms, while LoRA takes approximately 969ms). This is because the bridging module of the present invention performs a computation only once outside the LLM, while LoRA adds computational overhead at each layer of the LLM.

[0052] Economic Benefits (Training Cost and Scalability): The trainable parameters (bridging module, approximately 180.80M) of this invention remain constant and do not increase with the size of the LLM. In contrast, the number of parameters in LoRA increases proportionally with the size of the LLM. This means that for future larger-scale LLMs (such as 70B or larger), the adaptation cost (relative to the number of parameters) of this invention will be far lower than that of LoRA, demonstrating extremely high economic efficiency and scalability.

Claims

1. A method for cross-modal semantic bridging of code graphs and large language models, characterized in that, include: Convert the raw code text into a code attribute graph; The code attribute graph is input into the code graph encoder of the trained bridging system to obtain the code graph vector; The original code text and code graph vector are input into the bridging module of the trained bridging system to obtain text-graph embedding vectors; The text-graph embedding vector is projected into a soft cue vector through a large model projection layer; The input vector is obtained by concatenating the token embeddings of the soft prompt vector, the original code text, and the natural language instructions. The input vector is fed into the pre-trained LLM to obtain the answer corresponding to the natural language instruction.

2. The method according to claim 1, characterized in that, The code graph encoder is a A graph neural network with layers, where the update operation of each layer includes an attention mechanism modulated by edge features; where nodes For neighboring nodes The attention coefficient is: In the formula, For the first Layer nodes For neighboring nodes Attention coefficient; , , The first The query weight matrix, key weight matrix, and value weight matrix of the layer; For the first Layer nodes Node characteristics; For the first Layer nodes Node characteristics; As edge features, The dimension of the key; Aggregating nodes using attention mechanisms The node obtained after the neighbor node is represented as: In the formula, For the aggregated nodes Node characteristics; , All are learnable weight matrices. For nodes The set of neighbors.

3. The method according to claim 2, characterized in that, The bridging module is one The transformer consists of layers, each layer comprising a text encoder, a shared self-attention mechanism, a cross-attention mechanism, and a feedforward network connected in sequence.

4. The method according to claim 3, characterized in that, The specific method for obtaining the text-graph embedding vector is as follows: The original code text is input into the text encoder in the bridging module to obtain the code text embedding vector; By using a shared self-attention mechanism, information exchange is achieved between the code text embedding vector and the query vector, resulting in a query vector that integrates text information. Using code graph vectors as keys and values, and query vectors after fusing text information as queries, the keys, values, and queries are input into a cross-attention mechanism to obtain vectors injected with graph structure information. The query vector after fusing text information and the vector injected with graph structure information are input into the feedforward network to obtain the final output of this layer. go through After the above processing, the final text-graph embedding vector is output.

5. The method according to claim 4, characterized in that, The pre-trained code graph encoder, the pre-trained bridging module, and the large model projection layer constitute the bridging system; training the code graph encoder and training the bridging module are executed independently of each other.

6. The method according to claim 5, characterized in that, The training process of the code graph encoder is as follows: Randomly obtain code attribute graphs; The code attribute graph is randomly augmented twice to obtain the first augmented sample and the second augmented sample. The first and second augmented samples are input into the code graph encoder to obtain the first and second code graph vectors. By reducing the InfoNCE loss value, the distance between the first code graph vector and the second code graph vector of the same code attribute graph is brought closer, while the distance between different code attribute graphs is widened, until the InfoNCE loss value no longer decreases. Randomly sample node pairs from the code attribute graph and input them into the code graph encoder; output the edge type between node pairs. The ability of the code graph encoder to identify edge types is optimized by reducing the cross-entropy loss value until the cross-entropy loss value no longer decreases. The total training loss of the code graph encoder is: In the formula, This represents the total training loss value of the code graph encoder. For InfoNCE loss weights, For the InfoNCE loss function, For cross-entropy loss weights, This is the cross-entropy loss function.

7. The method according to claim 5, characterized in that, The bridging module is trained based on three complementary loss functions, including: Image-text comparison loss function: In the formula, This represents the loss value for image-text comparison. express loss function Represents a text-graph embedding vector. Embed vectors for code text. Learnable temperature; Image-text matching loss function: In the formula, For image-text matching loss value, For the sample size, For the first A true value, For the first One predicted value, Graph-guided text generation loss function: In the formula, To guide the generation of loss values ​​for the text, For the first The first sample One token, Indicates the first The text-graph embedding vectors corresponding to each sample; The total training loss of the bridging module is: In the formula, This represents the total training loss of the bridging module.

8. The method according to claim 5, characterized in that, The specific training method for the bridging system is as follows: Obtain training samples, including raw code text, natural language instructions, and standard answers; The original code text is converted into a code attribute map. The code attribute map and the original code text are then input into the bridging system to obtain the soft tooltip vector. The input vector is obtained by concatenating the token embeddings of the soft prompt vector, the original code text, and the natural language instructions. The input vector is fed into a pre-trained LLM to obtain the answer corresponding to the natural language instruction; The loss between the calculated answer to the natural language instruction and the standard answer is expressed as follows: In the formula, This represents the total training loss value of the bridging system. For all trainable parameters of the bridging system, As the standard answer, Conditional probability distributions generated for large language models For the input vector, The first in the standard answer sequence One token, The first in the standard answer sequence All tokens preceding this token; Calculate the total training loss of the bridging system with respect to The gradient is used to optimize the gradient using gradient descent. The trained bridging system is obtained; during the training of the bridging system, all parameters of the LLM are kept frozen and do not receive any gradient updates.

Images