Cross-modal adaptation fine tuning method and system based on double-branch network architecture

Abstract

The invention discloses a cross-modal adaptive fine tuning method and system based on a double-branch network architecture, and realizes cross-modal knowledge fusion and migration by combining the modeling capability of a graph neural network on code graph structures such as an abstract syntax tree, a control flow graph and the like and the powerful semantic understanding and generation capability of a large language model. A double-branch architecture is adopted, a graph structure and text input are coded respectively, and feature interaction is achieved through heterogeneous adaptation, structural decoupling and a mask attention mechanism. In a fine tuning stage, three-stage strategies are provided: basic fine tuning of structure perception, efficient parameter migration by combining Adapter, Prefix Tuning and LoRA, and language generation quality and robustness are optimized based on an unsupervised text reconstruction task. The training stage relates to structure sensing tasks such as node prediction, edge reconstruction and subgraph comparison, and the stability and generalization ability are improved in combination with course learning. The method is widely applicable to tasks such as code completion, question and answer and vulnerability detection, and has the advantages of strong structure perception, efficient cross-modal fusion, low training cost and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence and large language model, and particularly relates to a cross-modal adaptation fine-tuning method and system based on a double-branch network architecture. BACKGROUND

[0002] In recent years, with the wide application of large pre-training language models (LLMs) such as GPT, T5 and LLaMA, natural language processing (NLP) tasks have made breakthrough progress in multiple fields. However, there is still a bottleneck in processing structured or semi-structured data (such as knowledge graphs, abstract syntax trees AST, code call graphs, etc.). Such data often contains rich non-Euclidean structure information, and LLMs that rely solely on sequence modeling are difficult to effectively capture the topological dependencies between nodes, heterogeneous semantic topological structures and semantic associations, resulting in limited performance on such tasks.

[0003] In recent years, researchers have attempted to integrate GNN (Graph Neural Network) with LLM to leverage the collaborative capabilities of graph structure perception and language modeling. Some typical methods include: graph-enhanced input embedding, encoding graph structure information in sequence form and concatenating it to LLM input; graph-aware adapter, introducing a lightweight graph adaptation module in LLM to inject structured knowledge; multi-modal joint training: treating text and graph as two modalities, and improving generalization ability through joint optimization.

[0004] Although existing work has explored the integration of GNN and LLM, there are still challenges: weak structure representation: inputting graph structure in sequence form to language model may result in loss of structure information or insufficient representation; modal alignment difficulty: graph embedding and text embedding have significant differences in distribution, and direct concatenation may cause training instability or semantic drift; high cost of transfer and fine-tuning: current large models have a large number of parameters, and if efficient adaptation strategies are not used, it is often difficult to effectively transfer existing graph neural network knowledge in downstream tasks; high difficulty in modeling heterogeneous graphs: there are different types of nodes (variables, functions, etc.) and relationships (calls, dependencies, etc.) in code graphs, and traditional GNN structures are difficult to effectively capture their heterogeneous characteristics. SUMMARY

[0005] To address the above-mentioned deficiencies, the present application solves the technical problem of providing a cross-modal adaptation fine-tuning method and system based on a double-branch network architecture, which solves the problems of weak structure modeling, difficult modal fusion, high transfer cost and insufficient generalization ability of current large language models (LLMs) in processing structured and heterogeneous graph data, and perfectly realizes cross-modal and multi-task knowledge transfer and reasoning, especially suitable for code understanding, structured document analysis and other scenarios.

[0006] The first aspect of the present application provides a cross-modal adaptation fine-tuning method based on a double-branch network architecture, comprising:

[0007] S1: serializing text information and graph structure data by a model with a double-branch network architecture to obtain a text embedding matrix and a graph embedding matrix, and pairing the text embedding matrix and the graph embedding matrix.

[0008] S2: fusing the text information and the graph structure data according to the graph embedding matrix.

[0009] According to an embodiment of the present application, in the S1, the text information and the graph structure data are serialized by a model with a double-branch network architecture, and the model with the double-branch network architecture comprises a first pre-training module and a second pre-training module, specifically comprising: S101: serializing and outputting a text embedding matrix of the text information by the first pre-training module.

[0010] S102: serializing and outputting a graph embedding matrix of the graph structure data by the second pre-training module.

[0011] S103: feature concatenating the text embedding matrix and the graph embedding matrix, and pairing the text embedding matrix and the graph embedding matrix by a mask self-attention mechanism.

[0012] According to an embodiment of the present application, the first pre-training module and the second pre-training module have the following relationship: the first pre-training model adds a heterogeneous adaptation layer module to generate the second pre-training module.

[0013] According to an embodiment of the present application, in the S102, the graph embedding matrix of the graph structure data is serialized and outputted by the second pre-training module, comprising: traversing all nodes in the graph structure data, assigning a composite position number to the nodes, calculating the weight between the nodes, and generating the graph embedding matrix of the graph structure data.

[0014] According to an embodiment of the present application, in the S103, the text embedding matrix and the graph embedding matrix are feature concatenated, and the text embedding matrix and the graph embedding matrix are paired by a mask self-attention mechanism, comprising:

[0015] Suppose the text embedding matrix is a (Lxd) matrix, and the graph embedding matrix is a (Lxd) matrix; wherein L represents the number of text sequence words / graph nodes respectively; the text embedding matrix and the graph embedding matrix are concatenated into a (Lx2d) feature matrix; the feature matrix is randomly masked, and the text embedding matrix and the graph embedding matrix are paired.

[0016] The similarity of the text embedding vector and its corresponding graph embedding vector, the similarity of all graph embedding vectors, the similarity of the text embedding vector and its corresponding graph embedding vector, and the similarity of the text embedding vector and all graph embedding vectors are calculated, and the text embedding matrix and the graph embedding matrix are classified according to the calculation formula:

[0017]

[0018] wherein s(·) is a cosine similarity, is the embedding vector of the i-th text, is the embedding vector of the j-th graph node, τ is a temperature coefficient, k represents the number of graph nodes, and exp(·) represents summation.

[0019] According to an embodiment of the present application, the S2 fuses the text information and the graph structure data according to the graph embedding matrix, and comprises:

[0020] S201: The classified text embedding matrix and the graph embedding matrix are input into a language model together with the masked text after masking, and the probability that the masked text is predicted to be the original word is trained; nodes in the graph structure are randomly masked to a first threshold value, and the prediction type distribution of the masked nodes is trained; edges in the graph structure are randomly masked to a second threshold value, and the prediction probability of the type of the nodes and the edges is trained.

[0021] S202: An adapter module is added to map the aggregated graph embedding matrix into a semantic vector as input to a language training model.

[0022] According to an embodiment of the present application, the S2 fuses the text information and the graph structure data according to the graph embedding matrix, and further comprises: S203: training the model to perform code-related tasks not limited to one type through the composite input of natural language instructions, code context, and graph structure embedding.

[0023] According to an embodiment of the present application, the S203 trains the model to perform code-related tasks not limited to one type through the composite input of natural language instructions, code context, and graph structure embedding, and comprises:

[0024] A pre-trained model is obtained, and a code dataset related to a task is collected, each piece of data in the dataset comprising natural language instructions, code context, and graph structure embedding. The graph structure embedding is mapped to a language model through an adapter module.

[0025] The natural language instruction, code context and graph structure embedding are jointly encoded with a target output, and a model is trained to generate an output corresponding to a composite input, including: predicting a second output according to a first output, a natural language instruction and a code structure graph embedding, calculating a negative logarithmic loss of the second output and an answer, accumulating the negative logarithmic loss of all outputs to calculate a loss value, and training the model.

[0026] According to one embodiment of the present application, the method further comprises: according to the amount of training samples and the complexity of the samples, the samples are divided into first sample types, second sample types and third sample types, and different training strategies are set, including:

[0027] The first sample type does not change the double-branch network architecture model data, and the graph encoder and the adapter module are trained.

[0028] The second sample type changes the limited layer of the double-branch network architecture model data, and step S2 is executed.

[0029] The third sample type changes all the double-branch network architecture model data, executes step S2, and modifies the first threshold value, the second threshold value and the adapter related parameters.

[0030] The second aspect of the present application provides a cross-modal adaptation fine-tuning system based on a double-branch network architecture, which comprises: a collection module for serializing text information and graph structure data using a model of a double-branch network architecture to obtain a text embedding matrix and a graph embedding matrix, respectively, and pairing the text embedding matrix and the graph embedding matrix.

[0031] An adjustment module is used to fuse the text information and the graph structure data according to the graph embedding matrix.

[0032] According to one embodiment of the present application, the model of the double-branch network architecture of the collection module comprises: a first pre-training module and a second pre-training module, and the collection module comprises: a first collection unit for serializing the text embedding matrix using the first pre-training module.

[0033] A second collection unit is used to serialize the graph embedding matrix using the second pre-training module.

[0034] A splicing unit is used to splice the text embedding matrix and the graph embedding matrix, and pair the text embedding matrix and the graph embedding matrix through a mask self-attention mechanism.

[0035] According to one embodiment of the present application, the first pre-training module and the second pre-training module have the following relationship: the first pre-training model adds a heterogeneous adaptation layer module to generate the second pre-training module.

[0036] According to one embodiment of the application, the adjustment module comprises: a first reconstruction unit, which inputs the classified text embedding matrix and the graph embedding matrix, the masked text after masking, into a language model, trains the probability that the masked text is predicted as the original word; randomly masks the nodes in the graph structure of the first threshold value, trains the prediction type distribution of the masked nodes; randomly masks the edges in the graph structure of the second threshold value, trains the prediction probability of the edge type of the nodes.

[0037] A second reconstruction unit is configured to input the aggregated graph embedding matrix mapped as a semantic vector into a language training model through an adapter module.

[0038] According to one embodiment of the application, the adjustment module further comprises: a third reconstruction unit configured to train the model to perform code-related tasks not limited to one type through a composite input of natural language instructions, code context and graph structure embedding.

[0039] According to one embodiment of the application, the third reconstruction unit comprises: an input subunit configured to obtain a pre-trained model, collect a code dataset related to a task, and collect natural language instructions, code context and graph structure embedding in the dataset; a reconstruction subunit configured to map the graph structure embedding to a language model through an adapter module; and an output subunit configured to jointly encode the natural language instructions, code context and graph structure embedding with a target output, and train the model to generate an output corresponding to the composite input.

[0040] According to one embodiment of the application, the system further comprises a sample training module configured to divide samples into a first sample type, a second sample type and a third sample type according to a training sample amount and a sample complexity, and set different training strategies, including a first sample training unit configured to train a graph encoder and an adapter module without changing data of the double-branch network architecture model for the first sample type; a second sample training unit configured to change limited layers of data of the double-branch network architecture model for the second sample type, and jump to the adjustment module; a third sample training unit configured to change all data of the double-branch network architecture model for the third sample type, and jump to the adjustment module; and modify the first threshold value, the second threshold value and adapter-related parameters.

[0041] The third aspect of the application provides an intelligent device comprising a transmitter, a receiver, a memory and a processor; the memory is configured to store computer instructions; and the processor is configured to run the computer instructions stored in the memory to implement the above cross-modal adaptation fine-tuning method based on the double-branch network architecture.

[0042] The fourth aspect of the present application provides a storage medium, comprising: a readable storage medium and computer instructions, the computer instructions are stored in the readable storage medium; the computer instructions are used to realize the above cross-modal adaptation fine-tuning method based on the dual-branch network architecture.

[0043] The present application provides the beneficial effects: first, the system adopts a dual-branch architecture, respectively encodes the graph structure and the text input, and realizes feature interaction through heterogeneous adaptation, structure decoupling and mask attention mechanism. The underlying Transformer layers of the text branch and the graph branch share parameters. This means that in these two branches, the Transformer layers will use the same parameters, avoiding repeated calculation and wasting of computing resources.

[0044] Second, in the fine-tuning stage, a three-stage strategy is proposed, the first stage is the structure perception-based fine-tuning, after masking the text, node and edge relationships, the missing parts of the text, node and edge relationships are predicted and recovered again. Through this training method, the model can not only complete the text generation task in the case of missing information, but also ensure the quality and semantic consistency of the generated text. This means that the generated text not only needs to be correct in language, but also needs to be consistent with the original complete text in semantics to the greatest extent. At the same time, using an adaptive optimization algorithm, the learning rate can be dynamically adjusted during the fine-tuning process, so that the fine-tuning process is more efficient, and the problem of unstable training caused by too large learning rate is avoided.

[0045] In addition, according to the complexity and difficulty of different training samples, the samples are classified, and different training strategies are adopted for different sample categories. In the early stage, freezing the LLM can avoid a large amount of unnecessary calculation and focus on the training of downstream tasks (such as image encoders and adapters), thereby accelerating the convergence speed in the early stage. Effectively avoid the instability caused by too many LLM parameters in the early stage. Avoid overfitting, the introduction of medium complexity samples in the middle stage helps the model gradually adapt to a wider range of scenarios and tasks, improving the model's generalization ability. In particular, the top-level parameters of the language model are gradually unfrozen instead of being unfrozen all at once, avoiding the instability of training when all parameters are updated at the same time, while ensuring the gradual frame skipping of the model, thereby making the training process smoother. In the later stage, joint optimization of all parameters is performed, which can ensure that the model can fully utilize all the features it has learned to achieve optimal performance. At this time, the model has already fully understood the data, and appropriate global optimization can help the model further fine-tune. The later training makes the model gradually transition from simple basic feature learning to complex abstract feature representation, greatly improving the final performance of the model. BRIEF DESCRIPTION OF DRAWINGS

[0046] The accompanying drawings, which are incorporated herein and constitute part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, further serve to explain the principles of the present disclosure.

[0047] Figure 1 A flowchart of a cross-modal adaptation fine-tuning method based on a dual-branch network architecture disclosed in an embodiment of the present application;

[0048] Figure 2 A topology diagram of a cross-modal adaptation fine-tuning method based on a dual-branch network architecture disclosed in an embodiment of the present application;

[0049] Figure 3 A flowchart of a method for serializing text information and graph structure data using a dual-branch network architecture disclosed in an embodiment of the present application;

[0050] Figure 4 A flowchart of another cross-modal adaptation fine-tuning method based on a dual-branch network architecture disclosed in an embodiment of the present application;

[0051] Figure 5 A block diagram of a cross-modal adaptation fine-tuning system based on a dual-branch network architecture disclosed in an embodiment of the present application;

[0052] Figure 6 A flowchart of another cross-modal adaptation fine-tuning method based on a dual-branch network architecture disclosed in an embodiment of the present application.

[0053] Through the above-mentioned drawings, the explicit embodiments of the present disclosure have been shown, and will be described in more detail hereinafter. These drawings and written descriptions are not intended to limit the scope of the concept of the present disclosure by any means, but to illustrate the concept of the present disclosure to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0054] The exemplary embodiments will be described in detail herein with reference to the attached drawings. In the following description, the same numbers are used to indicate the same or similar components. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0055] The first aspect of the present application provides a cross-modal adaptation fine-tuning method based on a dual-branch network architecture, which combines the modeling capability of graph neural networks (GNN) for graph structure and the powerful capability of LLM in natural language understanding and generation, perfectly realizes cross-modal, multi-task knowledge transfer and reasoning, and is especially suitable for code understanding, structured document analysis and other scenarios.

[0056] As Figure 1As shown, the cross-modal adaptation fine-tuning method based on a dual-branch network architecture includes:

[0057] S1: The model using a dual-branch network architecture serializes text information and graph structure data into text embedding matrices and graph embedding matrices, respectively, and pairs the text embedding matrices and graph embedding matrices.

[0058] like Figure 2 As shown, a dual-branch Qwen architecture is adopted for efficient processing of serialized text and graph structure data. This dual-branch Qwen architecture divides the processing of text data and graph data into two independent branches through structural decoupling and cross-modal interaction mechanisms, ultimately achieving collaborative processing. Specifically, one of the branches is the text encoding branch, which is responsible for processing the input text data. The specific steps include: first, freezing the preprocessing Qwen model and directly using a pre-trained Qwen model to process the text data. In other words, the text branch uses a model that has been pre-trained on a large-scale dataset, avoiding training from scratch and saving training costs.

[0059] like Figure 3 As shown, the model using a dual-branch network architecture in S1 serializes text information and graph structure data, including: S101: using the first pre-training module to serialize and output the text embedding matrix of the text information.

[0060] The input text is first segmented, and then Qwen is used to generate context-dependent word embeddings. The text embeddings are then processed by extracting global semantic features through a standard Transformer layer, and finally outputting a text embedding matrix (with dimensions L×D, where L is the length of the text sequence and D is the embedding dimension).

[0061] S102: The second pre-trained module is used to serialize and output the graph embedding matrix of the graph structure data. The graph encoding branch is responsible for processing the graph structure data. This includes node features, edge types, and the adjacency matrix. The specific steps are as follows:

[0062] First, Qwen was modified by adding a heterogeneous adaptation layer to handle the complex structure of graph data. Graph data was mapped as pseudo-sequence input, which facilitated processing by the Transformer model.

[0063] Specifically, Qwen, as a pre-training model based on the encoder-decoder architecture, in which text data is usually processed through Transformer layers. But for graph data, the Qwen model needs to be modified to make it able to process graph structured data, especially heterogeneous graphs (graphs containing different types of nodes and edges). Make Qwen not only able to process text data, but also to process graph data. In order to adapt to graph data, Qwen needs to introduce some new mechanisms, such as a heterogeneous adaptation layer, in order to effectively process different types of nodes, edges and complex topological structures of graphs in the graph.

[0064] The main task of the heterogeneous adaptation layer is to convert the graph data (including node features, edge features, adjacency matrix, etc.) into a format that can be input into the Transformer model. In the process of processing graph data, the heterogeneous adaptation layer performs the following tasks: graph structure serialization, node and edge feature conversion, and position encoding expansion.

[0065] Regarding graph structure serialization, specifically, the graph data itself is a graph structure composed of nodes and edges, which needs to be converted into a "sequence" suitable for Transformer processing. The problem that graph data serialization usually faces is how to represent the complex relationship and structure information between nodes. A common approach is Meta-Path Sampling, which can generate "pseudo-sequences" from heterogeneous graphs, so that each node in the sequence represents its relationship with other nodes.

[0066] Regarding the implementation steps of Meta-Path Sampling, first define the meta-path, assuming the following node types and edge types, such as function (Function), class (Class), call statement (Call), variable (Variable), and file (File). Edge types include the inclusion relationship between functions and call statements (Function→Call), the structural relationship between classes and their member functions (Class→Function), the calling relationship between functions (Function→Function), the use relationship between functions and variables (Function→Variable), and the ownership relationship between files and classes or functions (File→Class / Function), etc. On this basis, we can define meta-paths with specific semantics, such as "Function→Call→Function" to represent the connection between two functions through a common call statement, or "File→Class→Function→Function" to represent the indirect association between functions in a file through class structure. By sampling these meta-paths, we can extract subgraphs with structural and semantic features to support subsequent code analysis, representation learning or downstream tasks.

[0067] The process of sampling is as follows: first, a starting node is selected from the graph, which can be randomly selected or selected according to task requirements. Then, starting from the starting node, the neighbor nodes are sampled according to the predefined meta-path rules, such as if the meta-path is function-call-function, starting from the starting node, first find a function node along the "function-call" edge, then find the next class node through the "function-ownership" edge, and so on.

[0068] At each sampling, the next node can be selected according to the sampling strategy, such as random sampling or weighted sampling. Weighted sampling can adjust the probability of sampling according to certain characteristics (such as the weight of the edge or the activity of the node).

[0069] In order to improve the diversity of meta-path sampling, multiple samplings can be performed, each from a different starting node or according to a different path, generating multiple "path samples". These samples can be used to train graph neural networks.

[0070] Finally, generate pseudo sequences, the output of meta-path sampling is a sequence of nodes, each sequence represents the relationship between different nodes in the graph. Through multiple sampling and path generation, a set of pseudo sequences can be obtained. These pseudo sequences are used as input for deep learning models such as Transformers, for processing of graph data.

[0071] For example, the sampled meta-path is function-call-function, and the pseudo sequence obtained by sampling may be:

[0072] [Function_1, Call_3, Function_5, Function_2], this sequence represents the relationship between Function_1 and Function 5 and Function_2 through calling.

[0073] In the process of meta-path sampling, the features of nodes and edges also need to be embedded and processed. For example, the features of nodes include variables, constants, code structure, etc. of functions (Function) and classes (Class). These features are usually mapped to low-dimensional vectors through embedding layers and used as input for the model.

[0074] Suppose there is a graph G containing different types of nodes and edges, the following is a pseudo code example of a simplified meta-path sampling process:

[0075]

[0076]

[0077] In the above code, the meta_path_sampling function samples nodes from the graph according to the given meta path. The sample_neighbors function samples the neighbors of the current node based on the node type and edge type.

[0078] It is clear that for a heterogeneous graph, the meta path sampling technique will sample according to the relationship between different types of nodes and edges in the graph. Through this process, the nodes of the graph are transformed into a series of node sequences, each corresponding to a specific meta path. A meta path is essentially a path of alternating node types and edge types that represents a certain high-order relationship between nodes. In the pseudo sequence, each node is mapped to a specific embedding vector. The embedding vector of a node is adjusted based on the characteristics of the node and its position in the meta path.

[0079] The pseudo sequence generated by meta path sampling is fed into the Transformer model. The input of each pseudo sequence includes the embedding of the node and the embedding of the edge, which can enhance the representation ability of the sequence through position encoding and node / edge features.

[0080] Regarding the transformation of node and edge features, specifically, each node and edge in the graph may have different features. In the heterogeneous adaptation layer, these features need to be mapped to a unified representation space, usually using an embedding layer or linear transformation, so that the features of each node and edge can be converted into a format suitable for model processing.

[0081] Regarding the extension of position encoding, nodes in graph data not only have types, but also involve the position or topology of the node in the graph. In order to adapt to the Transformer architecture, position encoding (usually used in sequence data) needs to be extended to consider the type, topological position, and other structural features of the node. This can usually be achieved by combining node type encoding and topological position encoding.

[0082] Due to the special nature of graph data, simply using standard position encoding (such as position encoding in Transformer) may not be sufficient to capture the complex relationships between nodes. Therefore, it is necessary to introduce composite position encoding, including node type encoding and topological position encoding, where node type encoding is used to represent the type of the node, so that different types of nodes in the graph have different semantics, such as "function" nodes and "class" nodes in code graphs, which represent different semantic types.

[0083] Among them, topological position encoding can help the model understand the position of the node in the graph, such as the neighbors of the node, the distance between nodes, etc. This position encoding can be calculated through the adjacency matrix of the graph or the topology of the graph.

[0084] This composite position coding, the representation of the node not only depends on its position, but also considers the node type and its topological position in the graph. It can capture the complex relationship between nodes more clearly and accurately.

[0085] In the traditional Transformer, additional mechanisms are introduced, such as the attention mechanism of heterogeneous graph, HGTHeterogeneous Graph Transformer) module, HGT module: In the Qwen model, the HGT (Heterogeneous Graph Transformer) module is introduced, and the heterogeneous graph attention mechanism is used to dynamically calculate the weight between nodes. In this way, when each node exchanges information with the neighbor nodes, it will not only calculate the weight according to the features of the neighbor nodes, but also according to the type of the edge between the neighbor nodes and itself. In Qwen, the HGT module is embedded, and the weight between nodes is dynamically calculated through the heterogeneous attention mechanism, which is specifically represented as follows:

[0086]

[0087] wherein, is a learnable mapping function of the relationship type r ij , which realizes the joint modeling of node type and edge type.

[0088] S103: Feature splicing the text embedding matrix and the graph embedding matrix, through the mask self-attention mechanism,

[0089] pair the text embedding matrix and the graph embedding matrix.

[0090] In the double-branch output layer, a probability mask attention layer is introduced to control the cross-modal interaction granularity through a random mask strategy. Specifically, first, the features are spliced, and the text embedding vector H t is spliced with the graph embedding vector to form Then a binary mask matrix M∈{0,1} (L+N)×(L+N) is generated based on the Bernoulli distribution, and the mask probability P is inversely proportional to the modal similarity. Perform mask self-attention on H:

[0091]

[0092] where Q is the query matrix, K is the key matrix, and V is the value matrix. d K is the dimension of the key vector, which is used as a scaling factor to prevent the value of the dot product from being too large, causing the softmax gradient to disappear.

[0093] The embedding vectors in the foregoing are a method of converting input data (such as text, graph nodes, or images, etc.) into low-dimensional continuous vectors. The embedding vectors capture the feature and semantic information of the data. The goal of embedding is to convert high-dimensional or discrete data (such as vocabulary, graph nodes) into a low-dimensional vector, which can mathematically represent the similarity or relationship between data.

[0094] Text embedding vectors are a method of representing the semantic information of text (such as sentences, paragraphs, words, etc.) by mapping them into a continuous vector space. Deep learning models are used to convert words in context into embedding vectors. The role of text embedding is to make semantically similar texts closer in the embedding space. For example, the words "cat" and "dog" are semantically related, so their embedding vectors should be relatively close in the vector space.

[0095] Graph embedding vectors are a method of mapping the structural information of nodes, edges, or entire graphs in a graph into a low-dimensional vector space. The embedding vector of each node contains the features of the node and its relationship with other nodes in the graph. For example, in a code graph, nodes can represent functions, and edges can represent the calling relationship between functions. Graph embedding methods (such as Node2Vec, GraphSAGE, DeepWalk) can generate node embeddings based on the relationship between nodes and the structure of the graph.

[0096] For example, the embedding vector of the node function Function can be [g1, g2,..., gD], where g1, g2,..., gD represent the features of the node and its position in the graph structure.

[0097] In multi-modal learning, the model usually needs to process feature information from different sources simultaneously. For example, in this case, there is text data (text vectors obtained through text embedding) and graph data (graph vectors obtained through graph embedding). In order to enable the model to consider both text information and graph structure information, these feature representations are usually concatenated into a unified vector.

[0098] Concatenation refers to connecting the text embedding vector and the graph embedding vector into a larger vector. The specific operation is:

[0099] Suppose the text data is converted into a vector by some method, in the form of [t1, t2,..., tD], where D is the dimension of the text embedding. Through graph embedding methods (such as Node2Vec, GraphSAGE, etc.), the embedding vector of a node in the graph is obtained, in the form of [g1, g2,..., gD].

[0100] After concatenating the two vectors, a new embedding vector is obtained, in the form of:

[0101] [t1,t2,...,tD,g1,g2,...,gD], so that the concatenated vector contains information of both text and graph. Its dimension is 2D, i.e., the embedding dimension of text and graph added together.

[0102] The purpose of the concatenation operation is to fuse information from different modalities (here, text and graph) into a common representation. Through concatenation, the model can learn with two different types of data. After fusing these two pieces of information together, the model can consider both text and graph features for comprehensive reasoning or classification tasks.

[0103] Masked Self-Attention is performed on the total embedding vector after concatenation. By introducing a probability mask, it can control which features will participate in the calculation of self-attention and which features will be "ignored".

[0104] The probability mask is generated based on the Bernoulli distribution, which means that each feature has a certain probability of being "masked" or ignored in attention calculation. The probability of the mask is inversely proportional to the modality similarity, which means that the modality with high similarity (for example, the similarity between text and graph) is more likely to be focused on, and the less relevant part may be masked.

[0105] In this way, the model can flexibly control the interaction granularity and importance of cross-modal according to the modality similarity and mask strategy.

[0106] Forcing the model to focus on the cross-modal high correlation feature area. Alignment optimization goal: on the basis of cross-entropy loss, increase the modality alignment loss function based on contrastive learning:

[0107]

[0108] where s(·) is the cosine similarity, is the embedding vector of the i-th text (e.g., obtained by Transformer). is the embedding vector of the j-th graph node (e.g., produced by the graph neural network). τ is the temperature coefficient, used to control the "sharpness" of the distribution in the contrastive loss function. The numerator part is the similarity between the i-th text node and the corresponding graph node . The similarity between the i-th text and its corresponding graph node (positive sample) is represented by exponential mapping and dividing by the temperature coefficient. Through exponential transformation and temperature adjustment, the similarity is more sensitive or smooth.

[0109] The denominator part is the similarity between all k graph nodes and the text node The sum of the similarity index, denoted as "negative samples", is used for normalization. The denominator is the sum of exp(s(·) / τ) for all graph nodes, which means that all graph nodes are considered as candidates to calculate the similarity with the text node.

[0110] The logarithm and negative sign indicate that this is a cross-entropy loss in contrastive learning, as simplified in equation (4):

[0111]

[0112] In essence, this loss form of L is a cross-entropy classification goal that "maximizes the similarity of positive samples and minimizes the similarity of negative samples".

[0113] This makes the text node close to the corresponding graph node in the embedding space (maximizes the similarity), while maintaining a distance from other graph nodes. That is, the truly paired graph-text embeddings are closer in the vector space, while other unrelated pairs remain far apart. Simply put, the text "cat" and the picture "cat" node should have similar embedding vectors.

[0114] The text branch retains the original Qwen capability, and the graph branch realizes heterogeneous relationship reasoning through HGT. The two branches share the underlying Transformer parameters to reduce computational overhead. The text and graph branches share parameters at the underlying Transformer layer, which reduces the number of parameters and improves learning efficiency and uniformity.

[0115] Cross-entropy loss function L CE , used for classification tasks, if the model is to determine which graph node a piece of text corresponds to, cross-entropy can be used to train it to output the correct class (graph node). The formula is simplified as follows:

[0116]

[0117] where y i is the true label, and p i is the model's predicted probability.

[0118] On the basis of the cross-entropy loss, a modal alignment loss is added, and the two losses are trained together, as shown in equation (6):

[0119] L total = L CE + λ·L (6)

[0120] where L CE helps the model to correctly classify / perform the main task, and L helps the model to establish alignment relationships between different modalities. λ is a weight coefficient that controls the importance of the two.

[0121] The text branch still retains the ability of the Qwen pre-training model, meaning that the text branch will still use the powerful language understanding ability of the Qwen model for processing. The graph branch uses an improved version of the Qwen model and processes heterogeneous graph data through a **HGT (Heterogeneous Graph Transformer)** module. The task of this part is to infer the heterogeneous relationships in the graph, that is, the mutual relationship between different types of nodes and edges, through the HGT module.

[0122] In order to reduce the computational overhead, the underlying Transformer layers of the text branch and the graph branch share parameters. This means that in both branches, the Transformer layers that compute images and text will use the same parameters, avoiding redundant computation and wasting computing resources.

[0123] The cross-modal alignment mechanism is used to enhance the fusion of text and image data. In this framework, the PMA module controls the interaction granularity of text and image features through the probability mask attention mechanism; at the same time, the modal alignment loss is added on the basis of the cross-entropy loss, and the consistency of text and image in the embedding space is ensured through the contrastive learning method; the model not only learns the classification task, but also makes the representations of different modalities such as text and graph / image closer in semantics through contrastive learning, thereby improving the overall effect, especially the cross-modal understanding ability. The model can learn more fine-grained alignment of text and image features during training, and through the sharing of Transformer parameters, the computational overhead is reduced.

[0124] S102: Migration and fine-tuning of large models. Transfer learning refers to applying a model that has been trained on certain tasks (pre-trained model) to new tasks, and fine-tuning refers to further adjusting the model according to new data.

[0125] The large model migration and fine-tuning module proposed by the present application aims to deeply integrate the structured knowledge contained in the pre-trained graph neural network (GNN) and the large language model (such as LLaMA, Qwen), thereby significantly improving the model's structural understanding and semantic expression ability in tasks such as code generation, automatic completion, and question answering. To realize the docking of feature spaces between GNN and LLM, a learnable adapter mechanism (Adapter) is introduced to fuse the structural representation output by the graph encoder with the contextual semantics of the language model. The adapter module is inserted into the multi-layer structure of the LLM, while maintaining the original expression ability of the language model, enhancing its perception and reasoning ability for heterogeneous graph structures.

[0126] Specifically, the entire migration and fine-tuning process is divided into three stages, which are optimized from three aspects of structure alignment, parameter adaptation, and instruction fine-tuning.

[0127] S201: Embed the classified text matrix and graph embedding matrix into the language model as input, train the probability of predicting the original word from the masked text; randomly mask the nodes in the graph structure of the first threshold, train the prediction type distribution of the masked nodes; randomly mask the edges in the graph structure of the second threshold, train the prediction probability of the edge type of the nodes.

[0128] First, structure alignment, this stage takes large-scale source code and its natural language description as input, adopts mask language modeling (MLM) task as the main training target, and learns text reconstruction combined with graph structure information. On the basis of freezing the main parameters of the language model, the model restores by masking part of the text segment and introducing structure embedding generated by the graph encoder, improving the graph-text collaborative modeling capability. In addition, this stage can introduce structure supervision tasks, such as masking nodes in abstract syntax tree (AST) and predicting node types; or randomly removing some edges to train the model to identify edge relationship types, thereby assisting the graph encoder to learn stronger structure semantic representation capability.

[0129] Specifically, aligning two different data (such as code and language description) can achieve better understanding and generation. Through the mask language modeling technology, part of the input text (usually words) is randomly masked, and the model is trained to predict the masked part according to the context. In this way, the model learns the semantics and structure in the text.

[0130] In the model training process, in addition to the language model of the text, the information of the graph structure is also combined. For example, the graph structure of the abstract syntax tree (AST) represents the syntax structure of the program. By introducing the embedding of the graph structure, the model can better understand the structure and syntax of the code.

[0131] In the text reconstruction task, the model needs to use the combination of language and graph to restore the masked text. Specifically, the masked text segment is restored by introducing the structure embedding of the graph encoder. This can make the model better understand the structure of the code and improve the ability of graph-text collaborative modeling.

[0132] The task of text reconstruction learning is: given a natural language description X = {x1, x2,... x L}, randomly mask part of it to get the masked input The graph structure encoder extracts the structure context embedding vector h graph of the text, and inputs it into the decoder together with the masked text. The model input includes graph structure embedding and masked text. The training target is to predict the original word t ithe probability of the model, and minimize the reconstruction error. The model generates complete text from the missing part of the language information using graph structure embedding and context, thereby improving the quality of generation and semantic consistency.

[0133] Specifically, let the input code text be Then its corresponding text embedding is extracted by the language model as:

[0134]

[0135] where h is the hidden dimension, H text is the output of the text branch.

[0136] At the same time, the code is parsed into a graph structure G = (V, E), where V is the node set and E is the edge set. The node feature output by the l-th layer graph encoder is :

[0137]

[0138] where d is the output dimension of the graph encoder (such as 768 dimensions), and the system aggregates the feature vectors of its node set {v1, v2, …, v k} into a context vector:

[0139]

[0140] where h graph is the structural context embedding, is the l-th layer node v i graph encoder output feature.

[0141] The total training target of the structure alignment stage consists of three sub-tasks: text reconstruction, node type prediction, and edge relationship reconstruction. The overall loss function is represented as follows:

[0142]

[0143] where α and β are hyperparameters, and the specific values are determined according to the training situation. L text is the text reconstruction loss, L node is the node type prediction loss, and L edge is the edge relationship prediction loss.

[0144] Text reconstruction, given the input sequence masked, the goal is to predict the original word x graph , i , as the global text representation, and its loss function is:

[0145]

[0146] Among them, h f This indicates fusion semantics, representing And graph structure h graph A unified vector, where M represents the set of masked word positions, x i The original words are masked out. In equation (12), a multilayer perceptron (MLP) is used to concatenate the text information with the graph structure context vector to generate a global text representation vector h. f The vector in the feature space R d middle.

[0147] In equation (13), x represents the model loss function, which means that by maximizing the word x... i Given a masked input sequence and global text representation vector h f The conditional probability is used for text reconstruction. The loss function is applied to all masked words x. i The summation of the log-likelihood functions. This method attempts to recover the original content of the text during training and optimizes the model by minimizing the negative logarithm of the predicted word probabilities.

[0148] For each masked position i, we calculate the word x predicted by the model for that position. i probability Then take its logarithm, sum the log probabilities of all mask words, and finally take the negative value. The goal of this process is to minimize the loss function, that is, to maximize the accuracy of the model's predictions.

[0149] Indicates that in a known input sequence and global text representation h f Under the condition of predicting word x i The probability. Loss function L text The goal is to make this probability as high as possible, that is, to hope that the words predicted by the model are as consistent as possible with the actual words.

[0150] The core idea of ​​optimization is to recover the original text by maximizing the conditional probability. This involves predicting the known parts of the text (the masked parts) to recover the original complete text.

[0151] Specifically, the input sequence to be masked is determined during training. This means that some words are masked, and the model is unaware of these masked words. This is to allow the model to predict and recover these masked parts using contextual information.

[0152] In addition to the context of the text, the model uses graph structure context vectors, which represent structural information related to the text (knowledge graph, social graph, etc.), to help the model understand the entities and relationships involved in the text, and to help predict the masked vocabulary.

[0153] During training, the model adjusts the parameters to optimize this loss function, so that it can predict more accurate words at each masked position. By minimizing the negative log probability, the model ultimately learns to effectively recover the masked part based on the context information and graph structure context, thereby recovering the complete text.

[0154] Node type prediction, in a graph structure (such as AST), randomly masks part of the node, and asks the model to predict its syntax type. For each masked node v∈V mask , whose corresponding token index is i v , the loss function is defined as:

[0155]

[0156] where V mask is the set of masked nodes, y v is the true syntax label of node v, and the text embedding corresponding to the node is extracted from the language model P(y v |·) is the node type distribution predicted by the graph encoder.

[0157] Edge relationship prediction loss: randomly remove part of the edges in the graph, and train the model to predict the type of the edge based on the node representation. The loss function is:

[0158]

[0159] where E mask is the set of masked edges, r uv represents the true type label of edge (u,v), h u, h v is the node representation output by the graph encoder, and are semantic vectors output by the language model, corresponding to token positions i u and i v , P(r uv |·) is the predicted probability distribution of edge type. Equations (16) and (17) represent the structure representation h u and the text representation are concatenated and fed into a multi-layer perceptron (MLP) to obtain new and

[0160] In formula (18), The model predicts the probability of edge types according to the representation of node pairs.

[0161] Through this training method, the model not only can complete the text generation task in the case of missing information, but also can guarantee the quality and semantic consistency of the generated text. This means that the generated text not only needs to be correct in language, but also should be consistent with the original complete text in semantics.

[0162] S202: Parameter fine-tuning. Join the adapter module to map the aggregated graph embedding matrix to a semantic vector as input to the language training model.

[0163] In this phase, parameter efficient fine-tuning technologies such as Adapter, Prefix Tuning, and LoRA (Low-Rank Adaptation) are introduced. By inserting a small-scale trainable parameter module between the graph encoder and the LLM, low-cost and efficient knowledge transfer and adaptation are achieved. While freezing most pre-training parameters, this phase enhances the model's adaptability to specific tasks, significantly reducing training overhead and mitigating overfitting risk.

[0164] One typical method is to insert a learnable mapping layer between the graph encoder and the LLM through the Adapter mechanism. Specifically, the node features output by the graph encoder are linearly transformed and added to the hidden state of the LLM as enhanced context input. For formula (9) and formula (10), the adapter is a lightweight neural network that maps the graph embedding h graph to the hidden space of the LLM. The typical implementation is a two-layer linear transformation plus an activation function:

[0165] h adapted = W2·ReLU(W1·h graph +b1) + b2 (19)

[0166] where h adapted is the output hidden dimension of the adapter, are learnable parameters, b1 and b2 are bias terms, ReLU is the activation function, and h is the hidden layer dimension of the LLM model. In this way, the graph embedding is mapped to a feature space compatible with the large language model, and then added or concatenated with the hidden state of the language model as enhanced context input.

[0167] The two weight matrices W1 and W2 in the formula are applied to different network layers. By fine-tuning the weights layer by layer, especially for deep networks, the problem of unstable training caused by excessive parameter space can be avoided. Fine-tuning the last few layers of the network on the pre-training basis and gradually adjusting the parameters of lower layers can effectively improve the efficiency of fine-tuning.

[0168] The use of ReLU activation function helps to prevent the problem of gradient vanishing, but in actual training, it is still possible to encounter the problem of gradient explosion. Using gradient clipping techniques can control the gradient, thereby ensuring the stability of the fine-tuning process. In addition, regularization (such as L2 regularization) can help prevent overfitting and ensure efficient fine-tuning.

[0169] In many deep learning tasks, part of the parameters of the pre-trained model can be frozen first, and only the last few layers are fine-tuned, and other parts of the model are gradually unfrozen. This approach not only improves the efficiency of fine-tuning, but also reduces the computational cost.

[0170] Using adaptive optimization algorithms, the learning rate can be dynamically adjusted during fine-tuning, making the fine-tuning process more efficient and avoiding the problem of unstable training caused by too large a learning rate.

[0171] By training the model jointly on multiple tasks and sharing some parameters during fine-tuning, it is possible to speed up convergence and improve the generalization ability of the model on different tasks. This multi-task learning strategy is very effective in complex graph data processing.

[0172] In terms of specific implementation, the KG-Adapter method can be used to insert special Adapter layers into the Transformer-based decoder structure. These Adapter layers encode graph structure information from both node-centric and relation-centric perspectives, helping the language model effectively capture the complex semantic structure in heterogeneous graphs. Prefix tuning means that the prefix vector is P = {p1, …, p m}, and the input sequence is expanded to Prefix input LLM model (such as Qwen) is added.

[0173] LoRA injection means that for the original weight matrix W ∈ R h×h , LoRA injects the following form of fine-tuning with a low-rank matrix:

[0174]

[0175] ΔW = A·B, A ∈ R h×r , B ∈ R r×h (21)

[0176] where r << h, i.e. much smaller than the dimension of the original matrix, reducing the parameter size. A, B are learnable parameter matrices representing low-rank decomposition. W remains frozen and does not participate in parameter updates. ΔW is a low-rank matrix that approximates the update of W.

[0177] The task of this stage is only text reconstruction, L stage2 text , using different fine-tuning methods to adjust the LLM model parameters without involving additional structural supervision tasks.

[0178] S203: Train the model to perform code-related tasks not limited to one type through the composite input of natural language instructions, code context, and graph structure embedding.

[0179] After completing the training of the two stages of structural reconstruction and parameter adaptation, to further improve the model's generalization ability and practical performance for specific tasks, the present invention introduces an instruction fine-tuning mechanism at this stage. This mechanism aims to guide the model to handle various code-related tasks in natural language format, such as code question answering, function completion, vulnerability detection, code generation, and rewriting. By organizing training data in a unified instruction format, the model can perform multi-task understanding and generation in a structured context.

[0180] In terms of input form, this stage model accepts a composite input composed of "natural language instruction + code context + graph structure embedding". The natural language instruction describes the task intent, such as "please complete the following function logic" or "please judge whether there is a security problem"; the code context is the code snippet to be processed; and the graph structure context is the structural vector h graph generated by the graph encoder (such as HGT) based on AST or CFG structure after pooling. This graph embedding is then mapped to the hidden space of the language model through the adapter module Adapter and inputted into the LLM along with the instruction and code.

[0181] During training, the present invention adopts a unified "input→response" modeling approach, which jointly encodes structured information and instruction input to train the model to generate the target response.

[0182] For example, given a function containing a missing statement and a completion request, the model needs to output the complete code that meets the semantic logic. This task uses an autoregressive language modeling objective to minimize the generation loss between the model output and the standard answer:

[0183] For a target output sequence y = {y1, y2, … y t}, the model predicts the current word y graph at each time step t based on the previously generated words y<t, natural language instructions, and code structure graph embedding h t ​The probability of the instruction being correct is calculated, and the negative logarithm of the probability is taken as the loss. Finally, the losses of all words are summed up, which is the total loss of the instruction fine-tuning stage. This stage supports the adaptation of multiple task formats, including but not limited to: code question answering (Code QA), function completion, code comment generation, vulnerability analysis, code rewriting, etc. Different tasks can be uniformly converted into standard format input through instruction templates, enabling the model to handle multiple tasks.

[0184] Specifically, S2031: Constructing an acquisition pre-training model, selecting a large language model as a base model (such as GPT, T5, CodeT5, CodeBERT, etc.). If the structure perception model, you can also choose a variant that supports structure input. At the same time, collect code datasets related to the task, including: function completion data, vulnerability detection data, code question pairs, code comments, etc. Each piece of data includes: original code snippets (code context), natural language instructions for the corresponding task (instruction templates), graph structure (generated through AST, CFG, etc.).

[0185] S2032: Use GNN to extract the graph embedding of AST / CFG, generate structure reconstruction and graph embedding, including using a graph encoder to extract graph structure embedding, using tools such as tree-sitter or PyTorch-AST to parse code into: AST (Abstract Syntax Tree) and / or CFG (Control Flow Graph), and input the structure into a graph neural network (such as HGT, GCN) for encoding.

[0186] Use average pooling and attention pooling methods to compress node vectors into vectors of uniform dimensions.

[0187] S2033: Construct an input of "natural language instruction + code context + graph embedding", each sample includes the above three parts, and the output is the target response that the model needs to generate. Graph embedding is converted into a hidden space matching the language model through an Adapter module.

[0188] adapted_graph=Adapter(pooled_graph)

[0189] final_input=tokenizer.encode(

[0190] instruction+code_context+adapted_graph)

[0191]

[0192] ​S2034: Autoregressive training using unified "input→response" approach, optimizing generation loss. Using autoregressive language modeling objectives (like GPT series), predicting the probability of the next word for each word, loss function is the negative log-likelihood function. Using standard "input→response" training paradigm, optimizing objectives:

[0193] loss = model(input_ids = final_input, labels = target_output).loss

[0194] loss.backward()

[0195] optimizer.step()

[0196] Batch training for each task data, loop for multiple epochs.

[0197] S2035: Design task templates and unify formats, implement multi-task mixed training. Unify the input formats of different tasks as shown in Table 1:

[0198] Table 1

[0199] Task Type Natural Language Instruction Function Completion "Please complete the following function logic" Vulnerability Detection "Please determine whether the function has a security sinkhole" Code Q&A "The following code and question, please answer" Code Rewrite "Please optimize this code and improve readability or efficiency"

[0200] S2036: Evaluate the model using the validation set and deploy it for practical applications. Use the validation set of different tasks to evaluate the accuracy of the generated (such as BLEU, CodeBLEU, Exact Match, F1, vulnerability detection accuracy, etc.). Deploy the trained model as an inference API that supports input natural language + code and outputs responses. It can also be used in IDE plugins, online code assistants, and other application scenarios.

[0201] As shown in Figure 4 , the cross-modal adaptation fine-tuning method based on the dual-branch network architecture of the application further comprises:

[0202] S3: According to the sample size and sample complexity of the training, the samples are divided into first sample type, second sample type and third sample type, and different training strategies are set, including: regarding node type prediction, 15% of the nodes in the AST graph are randomly masked, and the model is required to predict the type of the masked node (such as variable declaration, function call, etc.) based on the context graph structure, and the loss function is cross-entropy.

[0203] Regarding edge relationship reconstruction, 20% of the edges in the graph are randomly removed, and the model is required to predict the type of the missing edge (such as data dependency, control flow) based on the node features and the remaining topological structure, and the loss function is multi-class cross-entropy.

[0204] Text reconstruction learning task: Given a natural language description X = {x1, x2,... x L}, randomly mask a portion of it to get the masked input The graph structure encoder extracts the structural context embedding h graph corresponding to the text and sends it to the decoder along with the masked text to predict the probability of the original word t i at the masked position, minimizing the reconstruction error.

[0205] Adopt course learning strategy to improve the quality of graph-text alignment and model migration stability: According to the complexity of code snippets (such as AST depth, loop nesting level), the training data is divided into three levels: simple, medium and difficult;

[0206] For simple code, its AST depth is shallow, generally not more than 5 layers, the function body is relatively short, usually within 10 lines, and there is no nested structure. Such code usually only contains basic syntax elements such as variable declaration, assignment and simple print operation, without function call and control flow structure.

[0207] Entering the medium level, the complexity of the code increases, the AST depth is generally between 6 to 12, and the function body length also increases, usually within 30 lines. This level of code will contain basic control flow structures such as if statements, for or while loops, and may have one to two layers of control flow nesting, while it may involve a small amount of function calls and class structures. Cross-file dependencies are less, and the code structure is still clear, but it begins to show the characteristics of modularization and logical branching.

[0208] Difficult level code shows high complexity, with AST depth exceeding 12 layers, control flow nesting reaching three layers or more, and function body lines often exceeding 30 lines. Such code often contains recursive calls, exception handling, class inheritance relationships, and complex dependencies between multiple modules, and may even involve advanced programming techniques such as templates, generics, and concurrent calls. In addition, difficult level code may also involve cross-file calls and multi-language mixed calls (such as Python calling C modules), which are difficult to maintain and understand, and are suitable for complex systems and high-performance applications.

[0209] Specific progressive training: Early stage (1-10 rounds): only use simple samples, freeze LLM, optimize graph encoder and adapter. Middle stage (11-30 rounds): introduce medium samples, gradually unfreeze the top-level parameters of LLM. Late stage (31-50 rounds): load difficult samples, jointly optimize all trainable parameters. Dynamic data sampling: dynamically adjust the sampling probability of samples at each level according to the current performance of the model.

[0210] Task adaptation and dynamic inference, including: converting the original code (such as Python / Java) into structured graph data through syntax parsing (such as Tree-sitter). The nodes and edges of the graph represent the code syntax structure and logical dependencies, respectively, and each node is assigned an embedding vector containing the code semantics for use by downstream models (such as graph neural networks or multi-modal models). Code text parsing and graph conversion, including text processing and graph structure generation, text input processing includes: standardizing the original code text (such as Python, Java), including removing comments, unifying indentation formats, and using a pre-trained tokenizer.

[0211] Specifically, the original code is preprocessed to eliminate unnecessary interference factors and standardize the format, making it easier to further build a graph. A pre-trained code tokenizer (such as BPE, CodeBERT tokenizer, etc.) is used to convert the code into a token sequence, which lays the foundation for subsequent embedding and context representation.

[0212] Use static analysis tools (such as Tree-sitter) to parse the code text and generate an abstract syntax tree (AST), and further convert it into a heterogeneous graph structure. Use a static analysis tool like Tree-sitter to convert the source code into an abstract syntax tree (AST), which is a structured syntax representation of the code. The AST will be further converted into a "heterogeneous graph structure", that is, the graph has multiple types of nodes and multiple types of edges, not a simple graph.

[0213] Each node in the graph corresponds to a syntax element in the AST. The syntax element includes: function declaration, variable assignment, and control statement.

[0214] The edges between the nodes in the graph represent different types of dependency relationships, including: parent-child relationship: between parent and child nodes in the AST; data dependency: the relationship between the definition and use of a variable; control flow: such as if / else branching paths, loop paths, etc. These edges can represent the behavior paths and data flow patterns that may occur during code execution.

[0215] Each node is assigned an embedding, which represents its semantic information in the code context. Each node embedding contains code snippets, variable types, and context semantic vectors (initialized by pre-trained models), such as code snippets (e.g. int x = 0;), variable types (e.g. int, float, etc.), and context semantics. Context semantics can obtain relevant context vectors from pre-trained models (such as CodeBERT, CodeT5) to initialize the embedding. These node embeddings serve as input for neural networks for subsequent calculations.

[0216] Convert graph structure data (such as AST or other forms of graphs) into a serialized format that can be input into the model. Graph serialization and encoding.

[0217] Meta-path sampling: Extract key subgraphs from the AST graph according to task requirements (such as focusing on data flow paths for vulnerability detection), and expand them into pseudo-sequences according to meta-paths (such as "function call → parameter passing → return value").

[0218] Meta-path (Meta-path) is a specific path of nodes and edges in a graph, used to describe the relationship between entities in the graph.

[0219] For example, in the vulnerability detection task, we may focus on the key path: "function call → parameter passing → return value". Such a path can help the model identify potential vulnerabilities in the code.

[0220] Subgraph extraction: Extract specific subgraphs from the AST graph according to different task requirements (such as vulnerability detection, code completion), and process them to form pseudo-sequences dominated by meta-paths. These pseudo-sequences can represent the code logic relationships that need to be focused on in the task.

[0221] Heterogeneous position encoding: Generate a composite encoding vector for each node, including: Type Encoding: Map node types to vectors through a learnable embedding table; Topology Encoding: Generate position encoding based on the depth-first traversal order of nodes in the graph.

[0222] Nodes in the graph often have different attributes (types, positions, etc.), so we need to generate different composite encoding vectors for each node to effectively pass the structure and semantic information of the graph to the model.

[0223] After completing the construction of multi-modal input of graphs and text, the system performs parallel encoding and collaborative reasoning through a double-branch network to achieve deep fusion of graph structure and text semantics. For different task requirements (such as code completion, question answering, or vulnerability detection), a task adaptation module is introduced.

[0224] Code completion task. Use the Decoder with graph-aware pointer mechanism, which can refer to the structure information of nodes in the AST graph during generation. Given the current generation state, the model can dynamically retrieve the subgraph node embedding h v ∈H gAs the generation context. Code Q&A task inputs the user question (natural language) into the Encoder-Decoder architecture after fusing it with the retrieval subgraph, and the Decoder outputs the natural language answer or code fragment. Each sub-prediction branch of the vulnerability detection task extracts the representation from the hidden layer through the corresponding feature pool (such as MeanPool and MaxPool) and then performs MLP classification.

[0225] The second aspect of the present application provides a cross-modal adaptation fine-tuning system 50 based on a double-branch network mechanism, as shown in the figure, the system comprises: a collection module 501, which is used for serializing text information and graph structure data by using a double-branch network architecture model to obtain a text embedding matrix and a graph embedding matrix, and pairing the text embedding matrix and the graph embedding matrix. Figure 5

[0226] An adjustment module 502 is used for fusing the text information and the graph structure data according to the graph embedding matrix.

[0227] According to an embodiment of the present application, the double-branch network architecture model of the collection module comprises a first pre-training module and a second pre-training module, and the collection module comprises a first collection unit, which is used for serializing the text embedding matrix by using the first pre-training module.

[0228] A second collection unit is used for serializing the graph embedding matrix by using the second pre-training module.

[0229] A splicing unit is used for splicing the text embedding matrix and the graph embedding matrix, and pairing the text embedding matrix and the graph embedding matrix through a mask self-attention mechanism.

[0230] According to an embodiment of the present application, the first pre-training module and the second pre-training module have the following relationship: the first pre-training model adds a heterogeneous adaptation layer module to generate the second pre-training module.

[0231] According to an embodiment of the present application, the adjustment module comprises a first reconstruction unit, which inputs the classified text embedding matrix and graph embedding matrix, the masked text after masking, into a language model as input to train the probability that the masked text is predicted to be the original word; randomly masks the nodes in the graph structure with a first threshold value to train the prediction type distribution of the masked nodes; and randomly masks the edges in the graph structure with a second threshold value to train the prediction probability of the edge type of the nodes.

[0232] A second reconstruction unit is used for adding an adapter module to map the aggregated graph embedding matrix into a semantic vector as input to a language training model.

[0233] ​According to one embodiment of the present application, the adjustment module further comprises: a third reconstruction unit for training a model to perform code-related tasks not limited to one through the composite input of natural language instructions, code context, and graph structure embedding.

[0234] According to one embodiment of the present application, the third reconstruction unit comprises: an input subunit for obtaining a pre-trained model, collecting a code dataset related to a task, and the natural language instructions, code context, and graph structure embedding in the dataset; a reconstruction subunit for mapping the graph structure embedding to the language model through an adapter module; and an output subunit for jointly encoding the natural language instructions, code context, and graph structure embedding with a target output, and training the model to generate an output corresponding to the composite input.

[0235] According to one embodiment of the present application, as shown in Figure 6 The system further comprises a sample training module 503 for dividing samples into a first sample type, a second sample type, and a third sample type according to the amount of training samples and the complexity of samples, and setting different training strategies, including a first sample training unit for not changing the data of the double-branch network architecture model for the first sample type, training the graph encoder and the adapter module. A second sample training unit for changing limited layers of the double-branch network architecture model data for the second sample type and jumping to the adjustment module action. A third sample training unit for changing all the double-branch network architecture model data for the third sample type and jumping to the adjustment module action. Modify the first threshold value, the second threshold value, and the adapter-related parameters.

[0236] The third aspect of the present application provides an intelligent device, comprising a transmitter, a receiver, a memory and a processor; the memory is used to store computer instructions; the processor is used to run the computer instructions stored in the memory to realize the above cross-modal adaptation fine-tuning method based on double-branch network architecture.

[0237] The fourth aspect of the present application provides a storage medium, comprising: a readable storage medium and computer instructions, the computer instructions are stored in the readable storage medium; the computer instructions are used to realize the above cross-modal adaptation fine-tuning method based on double-branch network architecture.

[0238] The present application has the following beneficial effects: first, the system adopts a double-branch architecture, respectively encodes the graph structure and the text input, and realizes feature interaction through heterogeneous adaptation, structure decoupling and mask attention mechanism. The underlying Transformer layers of the text branch and the graph branch will share parameters. This means that in these two branches, the Transformer layers for computing images and texts will use the same parameters, avoiding repeated computation and wasting computing resources.

[0239] Secondly, in the fine-tuning stage, a three-stage strategy is proposed. In the first stage, the structure-aware fine-tuning is performed. After masking the text, nodes and edge relationships, the model predicts and restores the masked text, nodes and edge relationships. Through this training method, the model can not only complete the text generation task in the case of missing information, but also ensure the quality and semantic consistency of the generated text. This means that the generated text not only needs to be correct in language, but also needs to be consistent with the original complete text in semantics to the greatest extent. At the same time, by using an adaptive optimization algorithm, the learning rate can be dynamically adjusted during the fine-tuning process, so that the fine-tuning process is more efficient, and the problem of unstable training caused by too large learning rate is avoided.

[0240] In addition, according to the complexity and difficulty of different training samples, the samples are classified, and different training strategies are adopted for different sample categories. In the early stage, the frozen LLM can avoid unnecessary calculation and focus on the training of downstream tasks (such as image encoder and adapter), thereby accelerating the convergence speed in the early stage. The instability caused by too many LLM parameters in the early stage is effectively avoided. Overfitting is avoided, and the medium complexity samples introduced in the middle stage help the model gradually adapt to a wider range of scenarios and tasks, improving the generalization ability of the model. In particular, the top-level parameters of the language model are gradually unfrozen instead of being unfrozen at once, avoiding the instability of training when all parameters are updated at the same time, while ensuring the gradual frame skipping of the model, thereby making the training process smoother. In the later stage, joint optimization of all parameters is performed, which can ensure that the model can fully utilize all the learned features to achieve optimal performance. At this time, the model has fully understood the data, and appropriate global optimization can help the model further fine-tune. The later training makes the model gradually transition from simple basic feature learning to complex abstract feature representation, greatly improving the final performance of the model. Obviously, the above specific implementation cases are only examples for illustrating the application of the method, and are not limited to the implementation manner. For those skilled in the art, other different forms of changes and variations can be made on the basis of the above description to study other related problems. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0241] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, and the aforementioned program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the aforementioned storage medium includes ROM, RAM, magnetic disc or optical disc and various storage medium capable of storing program codes.

[0242] The embodiments of the electronic device described above are merely exemplary, and the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., may be located in one place, or may be distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0243] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be realized by means of software and necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, server, or network device, etc.) execute the method described in each embodiment or some part of the embodiment.

[0244] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the embodiments of the present application, and not to limit them; although the embodiments of the present application are described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

[0245] Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practice of the present disclosure. The present application is intended to cover any variations, uses, or adaptations of the present disclosure following, in general, the principles of the present disclosure and including such departures from the present disclosure that come within known or customary practice within the art to which the present disclosure pertains. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0246] It should be understood that the present disclosure is not limited to the precise structures described above and illustrated in the drawings and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the claims that follow.

Claims

1. A cross-modal adaptation fine-tuning method based on a dual-branch network architecture, characterized in that, The method includes: S1: The model using a dual-branch network architecture serializes text information and graph structure data into text embedding matrices and graph embedding matrices, respectively, and pairs the text embedding matrices and graph embedding matrices; S2: Based on the graph embedding matrix, fuse the text information with the graph structure data.

2. The method according to claim 1, characterized in that, The S1 model employs a dual-branch network architecture to serialize text information and graph structure data. The dual-branch network architecture model includes: a first pre-training module and a second pre-training module, specifically comprising: S101: The first pre-training module is used to serialize and output the text embedding matrix of the text information; S102: The graph embedding matrix of the graph structure data is serialized and output using the second pre-training module; S103: The text embedding matrix and the graph embedding matrix are concatenated using a feature-based self-attention mechanism to pair the text embedding matrix and the graph embedding matrix.

3. The method according to claim 2, characterized in that, The first pre-training module and the second pre-training module have the following relationship: The first pre-trained model is supplemented with a heterogeneous adaptation layer module to generate a second pre-trained module.

4. The method according to claim 3, characterized in that, In step S102, the second pre-training module is used to serialize and output the graph embedding matrix of the graph structure data, including: Traverse all nodes in the graph structure data, assign composite position numbers to the nodes, calculate the weights between nodes, and generate the graph embedding matrix of the graph structure data.

5. The method according to claim 2, characterized in that, In step S103, the features are concatenated with the text embedding matrix and the graph embedding matrix, and the text embedding matrix and the graph embedding matrix are paired using a mask self-attention mechanism, including: Assume the text embedding matrix is ​​an (L × d) matrix and the graph embedding matrix is ​​an (L × d) matrix; where L represents the number of words in the text sequence / the number of nodes in the graph, respectively; concatenate the text embedding matrix and the graph embedding matrix into a (L × 2d) feature matrix; randomly mask the feature matrix and pair the text embedding matrix and the graph embedding matrix; Calculate the similarity between the text embedding vector and its corresponding graph embedding vector, and the similarity between all graph embedding vectors. Classify the text embedding matrix and the graph embedding matrix using the similarity between the text embedding vector and its corresponding graph embedding vector, and the similarity between the text embedding vector and all graph embedding vectors. The calculation formula is as follows: in, Cosine similarity It is the first The embedding vector of each text. It is the first Each graph node embedding vector For temperature coefficient, Indicates the number of nodes in the graph. This indicates a summation.

6. The method according to claim 5, characterized in that, In step S2, the text information and the graph structure data are fused according to the graph embedding matrix, including: S201: Input the classified text embedding matrix and graph embedding matrix, along with the masked text, into the language model to train the probability that the masked text is predicted to be the original word; train the prediction type distribution of the nodes in the graph structure with the first threshold of random masking; train the prediction probability of the nodes for the edge type in the graph structure with the second threshold of random masking. S202: Add an adapter module to map the aggregated graph embedding matrix into semantic vectors, which are then used as input to the language training model.

7. The method according to claim 5, characterized in that, S2, which fuses the text information with the graph structure data according to the graph embedding matrix, further includes: S203: Train the model to perform code-related tasks, not limited to one type, using a combination of natural language instructions, code context, and graph structure embeddings as input.

8. The method according to claim 7, characterized in that, In step S203, the model is trained to perform code-related tasks, not limited to one type, using a composite input of natural language instructions, code context, and graph structure embeddings, including: Obtain a pre-trained model and collect a task-related code dataset, wherein each piece of data in the dataset includes natural language instructions, code context, and graph structure embeddings; The adapter module is used to embed and map the graph structure into the language model; The natural language instructions, code context, graph structure embedding, and target output are jointly encoded, and the trained model generates outputs corresponding to the composite inputs, including: The second output is predicted based on the first output, natural language instructions, and code structure graph embedding. The negative log loss between the second output and the answer is calculated. The negative log loss of all outputs is summed to calculate the loss value, and the model is trained.

9. The method according to claims 1 and 6, characterized in that, The method further includes: S3: Based on the training sample size and sample complexity, the samples are divided into three types: Type 1, Type 2, and Type 3, and different training strategies are set, including: The first sample type does not change the dual-branch network architecture model data, and trains the graph encoder and adapter module; For the second sample type, change the data of the dual-branch network architecture model of the limited layer and execute step S2; For the third sample type, change all the dual-branch network architecture model data, execute step S2, and modify the first threshold, the second threshold, and the adapter-related parameters.

10. A cross-modal adaptive fine-tuning system based on a dual-branch network mechanism, characterized in that, The system includes: The acquisition module is used to pair the text embedding matrix and the graph embedding matrix with dual-graph structure data, namely a text embedding matrix and a graph embedding matrix. The adjustment module is used to fuse the text information with the graph structure data according to the graph embedding matrix.

Images