A method for combining abstract syntax tree efficient parameter fine-tuning code model

Abstract

The application discloses a method for combining abstract syntax tree efficient parameter fine-tuning code model: collecting code snippets as training data and test data; automatically analyzing the code snippets, and converting the code snippets into abstract syntax tree AST containing code structure and syntax information; converting the abstract syntax tree into a graph data structure, and constructing a graph structure containing node and edge relationship; numerizing the graph structure through dimension reduction embedding technology, and generating abstract syntax tree embedding vector; taking the edge relationship of the graph structure and the embedding vector as the input of the graph neural network, learning the structured representation of the code snippet, and obtaining the constructed graph neural network; taking the constructed graph neural network as an adapter module, training the pre-trained code model using the training data, and performing efficient parameter fine-tuning in the training process to obtain the fine-tuned code model. The method can significantly improve the performance of the code model on various code tasks while reducing the parameter quantity of the fine-tuning.

Description

Technical Field

[0001] This invention relates to the application of deep learning models in code scenarios, specifically to a method for efficiently fine-tuning code models using abstract syntax trees. Background Technology

[0002] With the rapid development of deep learning technology, many large-scale pre-trained models have emerged in the code domain, such as CodeT5 and CodeBERT. These models have demonstrated powerful capabilities in code understanding and generation tasks, and can be used in various application scenarios such as code summarization, code generation, defect detection, quality analysis, code refactoring, and code translation. The emergence of code models has greatly improved the efficiency of software development and maintenance, and promoted the improvement of code quality and maintainability.

[0003] While current code models possess relatively general software development capabilities, in practical applications, fine-tuning is often required for specific tasks to further improve model adaptability. Traditional fine-tuning methods typically require updating all model parameters, which is time-consuming and requires significant training data and computational resources. To address this issue, researchers have proposed the concept of efficient parameter fine-tuning, which involves fixing most of the model's parameters and fine-tuning only a small subset to achieve rapid adaptation to different tasks. Efficient parameter fine-tuning methods have been widely applied in natural language processing, such as Prompt-Tuning, Prefix-Tuning, and LoRA. These methods can significantly reduce the data, time, and computational resources required for fine-tuning while maintaining high performance. For example, Chinese patent CN118132674A discloses a text information extraction method based on a large language model and efficient parameter fine-tuning: Utilizing the powerful text understanding and generalization capabilities of a large language model, different information extraction tasks are modeled as natural language inputs, and their output space and format are restricted. The model's output is then processed to form structured information. From a model training perspective, the efficient parameter fine-tuning method, with minimal parameter adjustments, allows the model to adapt to information extraction tasks while retaining its original language understanding and generalization capabilities. Compared to full-parameter training methods, it achieves the same or even better results, but requires significantly less computational resources and training time. From a generalization perspective, because the model's generalization ability is largely preserved, this method outperforms previous methods on low-resource and new-label scenarios, making it more suitable for practical applications.

[0004] However, existing efficient parameter fine-tuning methods are mainly aimed at natural language processing tasks and do not fully consider the special characteristics of code. Unlike natural language, code has strict grammatical rules and hierarchical structure. This structural information can be represented by an Abstract Syntax Tree (AST). Existing research has shown that introducing the code structural features contained in the AST into the code model can significantly improve the model's performance in code understanding and generation tasks. For example, Chinese Patent Publication No. CN113420296 A discloses a software defect prediction method and terminal based on graph convolutional neural networks, including: uniformly processing a dataset composed of code files, parsing it into an abstract syntax tree, adjusting the structure of the abstract syntax tree and retaining key information of the nodes; constructing a graph structure from the adjusted abstract syntax tree to build a training set; training a first prediction model, inputting the data to be predicted to obtain a first defect propensity result; training a second prediction model, inputting the data to be predicted to obtain a second defect propensity result; and averaging the two values ​​as the final defect prediction result.

[0005] Therefore, how to design an efficient parameter fine-tuning method that integrates code structure information to better improve the adaptability of existing code models in downstream tasks has become an urgent problem to be solved. Summary of the Invention

[0006] The purpose of this invention is to provide a method for efficiently fine-tuning code models by combining abstract syntax trees. This method can significantly improve the performance of code models on various code tasks while reducing the number of fine-tuning parameters.

[0007] This invention is achieved through the following technical solution:

[0008] A method for efficiently fine-tuning code models using abstract syntax trees (ASTs), the method comprising the following steps:

[0009] S1. Data Collection and Preprocessing: Collect code snippets as training and testing data;

[0010] S2. Automatic construction of abstract syntax trees: Automatically analyze code snippets and transform them into abstract syntax trees (ASTs) that contain code structure and syntax information;

[0011] S3. Transforming an abstract syntax tree into a graph structure: Transforming an abstract syntax tree into a graph data structure, constructing a graph structure containing node and edge relationships;

[0012] S4. Graph Structure Embedding: The graph structure is numericalized through dimensionality reduction embedding techniques to generate abstract syntax tree embedding vectors.

[0013] S5. Construction of Graph Neural Network: Using the edge relationships of the graph structure in step S3 and the embedding vectors in step S4 as inputs to the graph neural network, the structured representation of the code snippet is learned to obtain the constructed graph neural network.

[0014] S6. Fine-tuning of code model parameters: The constructed graph neural network is used as an adapter module. The pre-trained code model is trained using the training data from step S1. Efficient parameter fine-tuning is performed during the training process to obtain the fine-tuned code model.

[0015] Step S1 includes: organizing the training data and test data into different data formats according to different downstream tasks, and cleaning and preprocessing the training data and test data, wherein the preprocessing includes:

[0016] S1.1 Filter out invalid and duplicate code snippets;

[0017] S1.2 Remove redundant information from the code snippet;

[0018] S1.3 Format the code.

[0019] The data formats for the different tasks are as follows:

[0020] (1) Code summary task: <code snippet, corresponding natural language summary>;

[0021] (2) Code translation task: <Source code, target language code>;

[0022] (3) Vulnerability detection task: <code snippet, whether a specific vulnerability exists>.

[0023] The redundant information in step S1.2 includes comments, blank lines, etc.

[0024] The process of automatically constructing the abstract syntax tree in step S2 includes:

[0025] S2.1 Load the syntax file for a specific programming language;

[0026] S2.2. Use the syntax rules defined in the syntax file to parse code snippets and generate an abstract syntax tree;

[0027] S2.3 Merge and delete redundant branch nodes and leaf nodes in the abstract syntax tree to obtain a simplified abstract syntax tree.

[0028] In step S3, the method for transforming the abstract syntax tree into a graph structure is as follows:

[0029] S3.1. Use the root node of the abstract syntax tree as the starting node of the graph structure;

[0030] S3.2 Traversing the Abstract Syntax Tree:

[0031] S3.2.1 Map each node in the abstract syntax tree to a node in the graph structure and record the node type information;

[0032] S3.2.2 Map the relationship between each node in the abstract syntax tree and its parent and child nodes as directed edges in a graph structure, as edge relationships;

[0033] S3.3. Prune the graph structure to remove redundant nodes and directed edges, generating the final graph structure.

[0034] As a preferred method, transforming an abstract syntax tree into a graph structure mainly includes the following steps:

[0035] a) Define a graph structure G(N, E). Here, N represents the set of nodes in the graph, where each node corresponds to a node in the original tree structure. E represents the set of edges in the graph, where each edge represents a relationship between nodes. In the tree structure, each node (except the root node) has one and only one parent node and can have multiple child nodes.

[0036] b) Map each node in the tree structure to a node in the graph structure. If the tree structure has m nodes, then the transformed graph structure G will also contain m nodes, i.e., |N| = m.

[0037] c) Map each edge in the tree structure to a unidirectional edge in the graph structure. In the tree structure, each node (except the root node) has a parent node and may have multiple child nodes. For each parent-child node pair, convert it into an edge in the graph, pointing from the graph node corresponding to the parent node in the tree structure to the graph node corresponding to the child node in the tree structure. The set of edges E in the graph structure will contain all the edges corresponding to these parent-child relationships.

[0038] The graph structure embedding in step S4 includes:

[0039] S4.1 performs word segmentation on the node types and source code symbols in the graph structure;

[0040] S4.2 Expand the vocabulary of the pre-trained code model by adding new node types and symbols to the vocabulary to obtain the word vector matrix of the pre-trained code model;

[0041] S4.3 Initialize the node embedding vectors after word segmentation in the graph structure using the word vector matrix of the pre-trained code model;

[0042] S4.4 combines the node embedding vectors of each node after word segmentation into a node embedding vector by averaging or weighted summation.

[0043] In step S5, this invention constructs a Graph Neural Network (GNN) as an adapter module for the code model, explicitly incorporating the graph structure into the code model using various neighborhood aggregation strategies. The specific steps include: using the edge relationships of the graph structure from step S3 and the embedding vectors from step S4 as inputs to the GNN; secondly, the GNN aggregates the information of each node in the graph structure and its neighbors to reflect its position and the characteristics of its neighbors; finally, by stacking multiple GNN layers, the node representation can capture information from more distant neighbors, thereby enhancing the representational power of the graph neural network and further strengthening its ability to adapt to the code model.

[0044] As a further improvement, this invention considers that different graph neural networks have different representational capabilities and may be applicable to different downstream code tasks. This invention combines graph embedding with various graph neural networks to learn abstract code semantics. The main graph neural networks used are:

[0045] • Graph Convolutional Networks (GCNs): GCNs aggregate neighbor information by performing convolution operations directly on the graph. They use a normalized Laplacian matrix to achieve a weighted average of information, effectively capturing the local graph structure of nodes.

[0046] • Graph Attention Network (GAT): GAT introduces an attention mechanism to dynamically determine the importance of each neighbor when aggregating neighbor node information. This allows the model to more flexibly capture complex relationships between nodes.

[0047] • GraphSage: GraphSage is designed for large-scale graph data. It updates the representation of a target node by sampling a fixed-size set of neighbors and then aggregating information from these neighbors. This approach is both efficient and scalable.

[0048] The efficient parameter fine-tuning method in step S6 is as follows:

[0049] S6.1 Load the pre-trained code model;

[0050] S6.2. Use the graph neural network constructed in step S5 as an adapter module and connect it to the pre-trained code model through soft hints or prefix tuning.

[0051] S6.3 On the training data of downstream tasks, only fine-tune the adapter module parameters in the code model, and freeze the rest of the code model.

[0052] The soft-hint tuning method, also known as prompt tuning, specifically uses the output of the graph neural network described in step S5 as a prompt. This prompt tuning guides the output of the code model and downstream tasks. This strategy allows the model to more accurately adapt to different downstream task requirements, improving task specificity.

[0053] Preferably, in step S6.2, the soft hint fine-tuning method is as follows: insert the vector output of the graph neural network as a "soft hint" at the input of the pre-trained code model, and concatenate the "soft hint" with the text embedding vector of the source code fragment as the input of the encoder in the code model; in the fine-tuning stage, only the parameters of the graph neural network vector are updated, and the rest of the code model is frozen.

[0054] In this step, the same word vector matrix as in step S4.3 is used to encode the segmented source code fragments to obtain the text embedding vectors of the source code fragments.

[0055] Preferably, in step S6.2, the prefix fine-tuning is performed as follows: a learnable prefix vector is introduced into each multi-head attention layer of the encoder and decoder of the code model. This is achieved by concatenating the output of the graph embedding module as the prefix vector with the key-value matrix of the multi-head attention layer. In other words, this invention introduces prefix fine-tuning into the Transformer structure of the code model, using the output of the graph neural network as a learnable prefix vector to adjust the prefix weights of the KV matrix in each attention layer.

[0056] In step S6, the efficient parameter fine-tuning process is specifically manifested in the gradient backpropagation process, where only the gradient information of the adapter module is updated, without updating the gradient information of the code model. This parameter update method allows the model to efficiently adapt to different downstream code tasks, such as code summarization tasks, code translation tasks, and vulnerability detection tasks.

[0057] As a further improvement, for each different downstream code task, the present invention can train a specific code adapter through the dataset collected and processed in step S1 to improve the performance of the specified code task.

[0058] The present invention also provides an application of the code model obtained according to the above method in code tasks.

[0059] The code task is as follows:

[0060] Code summary task: Input code snippets into the model and decode them to generate corresponding natural language descriptions;

[0061] Vulnerability detection task: Input code snippets into the model, decode them to generate vulnerability types or the probability of vulnerability existence;

[0062] Code translation task: Input source code into the model and decode it to generate code in the target language.

[0063] The beneficial effects of this invention are as follows:

[0064] This invention proposes a method for efficient parameter fine-tuning of code models using abstract syntax trees (ASTs) in the code domain. Based on multiple open-source code task datasets, this method collects and categorizes different code task datasets, automatically generates ASTs for each dataset, and integrates these ASTs into the large code model using graph neural networks (GNNs) as adapters. The large model is then trained using efficient parameter fine-tuning, making it applicable to multiple downstream code application scenarios. This invention leverages the unique structural information within code text, introducing ASTs into the code model as adapter modules to fully mine and utilize the structured features of code, thus improving the model's understanding and representation capabilities. It provides a pluggable GNN adapter module framework that integrates with the code model, flexibly adapting to various graph neural networks to further meet the needs of multiple downstream code tasks. The efficient parameter fine-tuning strategy updates only a small number of adapter module parameters during the fine-tuning process, significantly reducing training resources and time costs while substantially improving the performance of the code model on downstream tasks, demonstrating significant theoretical and practical implications.

[0065] Compared to existing fine-tuning methods (such as prefix tuning), the efficient parameter tuning code model provided by this method demonstrates significant performance improvements across various programming languages. For example, in the code summarization task, measured by the BLEU4 metric, the prefix tuning method combined with AST proposed in this invention achieves performance improvements ranging from 0.72% to 4.36% in programming languages ​​such as Ruby, Javascript, Go, PHP, and Python. In other downstream code tasks, the efficient fine-tuning model proposed in this invention also shows performance improvements. In summary, the method for efficient parameter tuning code models combined with AST proposed in this invention fully utilizes the structured information of the code, introduces a flexible graph neural network in the form of an adapter, and employs an efficient tuning strategy. This significantly improves the performance of the code model across multiple downstream tasks while reducing resource overhead, demonstrating significant technical advantages and practical value. Attached Figure Description

[0066] Figure 1 A flowchart of an efficient parameter fine-tuning code model method combining abstract syntax trees provided by the present invention;

[0067] Figure 2The present invention provides a structural flowchart of a method for efficient parameter fine-tuning code models that combines abstract syntax trees;

[0068] Figure 3 This is an example diagram for converting source code into a simplified abstract syntax tree. Detailed Implementation

[0069] The specific technical solution of the present invention will be further described below with reference to the accompanying drawings:

[0070] This invention is achieved through the following technical solution: First, training and evaluation data are collected from open-source websites; then, an abstract syntax tree (AST) related to the dataset is constructed through an automated process; second, the AST is transformed into a graph structure; then, the graph structure is transformed into graph embeddings through an embedding vocabulary; simultaneously, a graph neural network model is constructed, and the graph neural network is integrated into the code model as an adapter module; finally, a large code model is trained through efficient parameter fine-tuning, and the adapter module is used to efficiently adapt to various code tasks, such as code summarization, code translation, and vulnerability detection. This method utilizes the structured information in the code and, with limited training resources, data resources, and time costs, quickly obtains a high-performance fine-tuned model, applicable to multiple downstream code application scenarios, including but not limited to code documentation generation, code translation, and vulnerability detection.

[0071] like Figure 1 and Figure 2 As shown in the flowchart, the method for efficient parameter fine-tuning of code models using abstract syntax trees provided in this embodiment includes the following steps:

[0072] S1: Organize the training and testing data into different data formats according to different downstream tasks, and clean and preprocess the training and testing data:

[0073] Preprocessing includes:

[0074] S1.1 Filter out invalid and duplicate code snippets;

[0075] S1.2 Remove redundant information (including comments, blank lines, etc.) from the code snippet;

[0076] S1.3 Format the code.

[0077] In this embodiment, the dataset needed for training and evaluating (testing) the code model is first collected from an open-source code platform, such as GitHub (one of the world's largest code hosting platforms). Data acquisition methods include crawling code snippets and utilizing existing datasets. After collecting and integrating the data into a raw dataset, data processing is performed. This processing includes filtering meaningless samples and removing meaningless characters. Finally, based on different downstream tasks, the collected dataset is categorized into code summary datasets, code translation datasets, and vulnerability detection datasets.

[0078] Each data entry in each dataset consists of two parts, with the following specific formats: for the code summary dataset, the data format is <code snippet, corresponding natural language summary>; for the code translation dataset, the data format is <source code, target language code>; and for the vulnerability detection dataset, the data format is <code snippet, whether a specific vulnerability exists>.

[0079] S2. Automatic Abstract Syntax Tree (AST) Construction: Automated analysis of code snippets, transforming them into an Abstract Syntax Tree (AST) containing code structure and syntax information.

[0080] S2.1 Load the syntax file for a specific programming language;

[0081] S2.2. Use the syntax rules defined in the syntax file to parse code snippets and generate an abstract syntax tree;

[0082] S2.3 Merge and delete redundant branch nodes and leaf nodes in the abstract syntax tree to obtain a simplified abstract syntax tree.

[0083] In this embodiment, the code portions of the collected dataset are transformed into an Abstract Syntax Tree (AST) using the tool Treesitter. Treesitter is an open-source generator tool that uses the syntax rules of programming languages ​​to generate ASTs. An AST displays the syntactic structure of the source code in a tree-like structure, where each node represents a part of the code, such as an expression, statement, or operator. This tree-like representation reveals deep structural information in the serialized code, enabling development tools and systems to understand and manipulate the code more intelligently. In this invention, the AST helps the model better understand the semantic information of the code. After being transformed into an AST, the AST is further parsed to generate a simplified version of the AST suitable for graph neural networks.

[0084] In step S2.3: Based on the above abstract syntax tree, measures are taken to remove redundant nodes in the syntax tree to generate a further simplified version of the abstract syntax tree. The generation steps of the simplified abstract syntax tree include the following three stages: First, leaf nodes that do not carry semantic information, such as semicolons, left parentheses, and right parentheses, are removed from the abstract syntax tree; for non-leaf nodes, only non-leaf nodes with strong structural information are retained, that is, only nodes containing program execution flow information and variable declaration information are retained, such as "Declaration", "Statement", "Call", and "Block" nodes; for the deleted non-leaf nodes, their parent nodes and child nodes are connected to form a new simplified abstract syntax tree. This simplified abstract syntax tree reduces the dimensionality of the graph neural network while retaining key semantic information.

[0085] S3. Transforming the Abstract Syntax Tree into a Graph Structure: This involves converting the abstract syntax tree into a graph data structure, constructing a graph structure that includes node and edge relationships.

[0086] The Treesitter AST is traversed using a breadth-first search, and a queue data structure is applied to store node information, including:

[0087] S3.1 Initialize the node list, edge list, and leaf node ID list.

[0088] S3.2. Enqueue the root node, and the elements in the queue are tuples (parent_id, node).

[0089] S3.3 For leaf nodes, obtain the corresponding source code and add it to the node list and leaf node ID list.

[0090] S3.4 For non-leaf nodes, add the node type to the node list.

[0091] S3.5 Add a bidirectional edge between the current node and its parent node to the edge list.

[0092] The above steps convert the source code into an AST representation that is easy to process and analyze. Based on this, nodes are encoded and truncated to meet the needs of further code analysis and processing tasks. Figure 3 An application example is presented, along with the source code for finding the maximum of two numbers. The corresponding AST structure generated by this method is also shown, including a list of nodes, an edge list, and a list of leaf node IDs, facilitating further code analysis.

[0093] S4. Graph Structure Embedding: Graph structures are numericalized using dimensionality reduction embedding techniques to generate abstract syntax tree embedding vectors.

[0094] S4.1 Perform word segmentation on the node types and source code symbols in the graph structure;

[0095] S4.2 Expand the vocabulary of the pre-trained code model by adding new node types and symbols to the vocabulary to obtain the word vector matrix of the pre-trained code model;

[0096] S4.3 Initialize the node embedding vectors after word segmentation in the graph structure using the word vector matrix of the pre-trained code model;

[0097] S4.4 Combine the node embedding vectors of each node after word segmentation into a node embedding vector by averaging or weighted summation.

[0098] This invention transforms each node in a graph structure, represented as a string, into a continuous, fixed-length vector. In this embodiment, the embedding process specifically includes the following steps:

[0099] a) Use the same tokenizer as the code model to decompose the variable names and keywords of the graph structure nodes into finer-grained subtokens.

[0100] b) Using the same embedding matrix as the code model, the sub-tokens described above are converted into vector form. This step encodes each sub-token as a fixed-length continuous vector, capturing its semantic and syntactic features in the code.

[0101] c) By summing the sub-token vectors within the same graph structure node, a synthetic embedding vector representing the original graph structure node is constructed. This synthetic embedding vector inherits information from all the sub-tokens that make it up, thus providing a comprehensive representation of the graph structure node for further code analysis and processing.

[0102] S5. Graph Neural Network Construction: Using the edge relationships of the graph structure in step S3 and the embedding vectors in step S4 as inputs to the graph neural network, the structured representation of the code snippet is learned to obtain the constructed graph neural network.

[0103] S5.1 Based on the graph structure in S4, use the embedding module in the code model to process the strings in the graph nodes, generate vectors as the embedding representations of the graph nodes, and use the embedding representations of all nodes and the edge relationships between nodes as the input of the graph neural network.

[0104] S5.2 Define the graph neural network encoder GraphEncoder; initialize all graph neural network layers, and determine the number of graph neural network layers N according to the configuration parameters; select the graph convolution type according to the configuration parameters, considering that different graph neural networks have different representation capabilities and may be suitable for different downstream code tasks. The optional types are GCN, GraphSAGE and GAT.

[0105] S5.3 For the first N-1 layers, the selected graph convolution type module is used to perform graph convolution operations on the node features and apply the ELU activation function; by aggregating the information of each node and its neighborhood in the graph structure, the position of the node in the graph structure and the features of its neighbors are reflected.

[0106] S5.4 For the last layer, the DenseDiffpool module is used to perform graph convolution operations on the node features, and the differential pooling method is used to pool the graph to generate the final virtual node representation.

[0107] S5.5 enhances the representational power of the graph neural network by stacking multiple graph neural network layers, enabling node representations to capture information from more distant neighborhoods, thereby further enhancing the graph neural network's ability to adapt to code models.

[0108] Through the above steps, the method provided by this invention embeds the graph structure using a large code model embedding module, encodes the input using a graph neural network, extracts node features through multi-layer graph convolution operations, and pools the graph using a differential pooling method to generate the final virtual node representation. By explicitly incorporating graph structure information into the code model and utilizing various neighborhood aggregation strategies, the ability of the graph neural network to adapt to the code model is enhanced.

[0109] S6. Code Model Parameter Fine-tuning: Using the constructed graph neural network as an adapter module, the pre-trained code model is trained using the training data from step S1. Efficient parameter fine-tuning is performed during training to obtain the fine-tuned code model.

[0110] S6.1 Loading pre-trained code models: Use the huggingface / transformer open-source framework to download and load pre-trained code models, such as CodeT5, CodeLlama, etc.

[0111] S6.2. Use the graph neural network constructed in step S5 as an adapter module and connect it to the pre-trained code model through soft hints or prefix fine-tuning.

[0112] • Use prompting to import the graph embedding module: embed the graph in the code.

[0113] This indicates that it is used as a soft suggestion and concatenated with the original code embedding as the final input to the model;

[0114] The output of the graph embedding module is added as a soft prompt to the encoder's input sequence. The graph embedding module is introduced using prefix tuning: in the encoder of the code model...

[0115] In each multi-head attention layer of the decoder, a learnable prefix vector is introduced; the output of the graph embedding module is used as the prefix vector and concatenated with the key-value matrix of the attention layer.

[0116] S6.3 On the training data of downstream tasks, only fine-tune the adapter module parameters in the code model, and freeze the rest of the code model.

[0117] In step S6.3:

[0118] Based on the task dataset collected and processed in the preceding steps, the code model incorporating the graph embedding module is fine-tuned. Specifically, this invention uses the AdamW optimization algorithm and sets the hyperparameters (learning rate 1e-5, batch size 16) for training.

[0119] The fine-tuning process employs an efficient parameter fine-tuning strategy to reduce the number of parameters that need to be updated. Specifically, all parameters of the pre-trained code model are frozen, and only the parameters of the graph embedding module are fine-tuned. Ultimately, the parameters fine-tuned represent only a small fraction (less than 5%) of all parameters. The parameter freezing is achieved by calculating only gradients during backpropagation without updating the parameters of the larger code model.

[0120] During training, efficient parameters are saved and module parameters are fine-tuned based on the results on the validation dataset.

[0121] S7. Inference Evaluation: After training, the adapter model parameters saved during the efficient parameter fine-tuning phase are used to perform inference and performance evaluation on the evaluation dataset (downstream task adaptation: the fine-tuned model is fine-tuned and evaluated on downstream tasks such as code summarization, vulnerability detection, and code translation to verify the model's adaptability to different tasks). The specific steps are as follows:

[0122] S7.1 Model Initialization and Loading: This embodiment first initializes a new model instance and loads the pre-trained code model parameters as the initial weights of the model backbone network; secondly, it loads the adapter module parameters learned in the efficient parameter fine-tuning stage, including the weights of the graph neural network and soft cues / prefix vectors; finally, it combines the adapter module with the pre-trained model backbone to construct a complete inference model.

[0123] S7.2 Model Inference: First, based on the task type, the evaluation dataset is transformed into an input format acceptable to the model: for code summarization tasks, code snippets are transformed into encoder input; for code translation tasks, source code is transformed into encoder input and target code into decoder output; for vulnerability detection tasks, code snippets are transformed into encoder input. Second, the constructed inference model is used to infer the evaluation data and decode the corresponding results: for code summarization tasks, the corresponding natural language summary is generated; for code translation tasks, the translated target code is generated; for vulnerability detection tasks, binary classification discriminant values ​​are generated (1 represents the existence of a vulnerability, 0 represents the absence of a vulnerability).

[0124] In this embodiment, the evaluation index is calculated as follows:

[0125] For code summarization tasks, the BLEU (Bilingual Evaluation Understudy) metric is used to measure the quality of generation by calculating the n-gram overlap between the generated natural language summary and the reference summary. BLEU-4 is commonly used as the evaluation metric. BLEU-4 considers the overlap between the generated summary and the reference summary across 1 to 4 consecutive word groups (unigram, bigram, trigram, and 4-gram).

[0126] For code translation tasks, the CodeBLEU metric is used. This metric, based on the traditional BLEU metric, incorporates structural information from the Abstract Syntax Tree (AST). By comparing the AST structural similarity and n-gram matching degree between the generated code and the reference code, it better measures the quality of code translation. CodeBLEU has been proven to outperform the traditional BLEU metric in code translation tasks.

[0127] For vulnerability detection tasks, accuracy, precision, recall, and F1 score are used as evaluation metrics.

[0128] Specifically, in this embodiment, the effectiveness of the present invention is evaluated on the CodeXGLUE dataset. The evaluation results are illustrated using a code summarization task as an example, as shown in Table 1.

[0129] The results show that this invention, by combining abstract syntax trees and using prefix tuning, efficiently fine-tunes the CodeT5-Base base model, enabling it to learn and retain code structure information from the source code, thus improving model performance. Compared to using general prefix tuning methods, this invention effectively improves code summarization tasks across different languages.

[0130] Table 1 shows the BLUE scores on the code summarization task for ordinary prefix tuning and prefix tuning combined with AST.

[0131]

[0132] Corresponding to the aforementioned embodiments of the method for efficiently fine-tuning code models using abstract syntax trees, the present invention also provides an apparatus for efficiently fine-tuning code models using abstract syntax trees, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement the method for efficiently fine-tuning code models using abstract syntax trees as described in the above embodiments.

[0133] The embodiments of the present invention, which combine an abstract syntax tree with an efficient parameter fine-tuning code model, can be applied to any device with data processing capabilities, such as a computer or other similar device. The device embodiments can be implemented in software, hardware, or a combination of both.

[0134] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0135] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0136] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the method for efficiently fine-tuning the code model using an abstract syntax tree as described in the above embodiments.

[0137] The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data-processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., mounted on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data-processing device. The computer-readable storage medium is used to store computer programs and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0138] It will be understood by those skilled in the art that the above description is merely a single example of the invention and is not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.

Claims

1. A method for efficiently fine-tuning code models using abstract syntax trees, characterized in that, The method includes the following steps: S1. Data Collection and Preprocessing: Collect code snippets as training and testing data; S2. Automatic construction of abstract syntax trees: Automatically analyze code snippets and transform them into abstract syntax trees (ASTs) that contain code structure and syntax information; S3. Transforming an abstract syntax tree into a graph structure: Transforming an abstract syntax tree into a graph data structure, constructing a graph structure containing node and edge relationships; S4. Graph Structure Embedding: This involves numericalizing graph structures using dimensionality reduction embedding techniques to generate abstract syntax tree embedding vectors; specifically including: S4.1 performs word segmentation on the node types and source code symbols in the graph structure; S4.2 Expand the vocabulary of the pre-trained code model by adding new node types and symbols to the vocabulary to obtain the word vector matrix of the pre-trained code model; S4.3 Initialize the node embedding vectors after word segmentation in the graph structure using the word vector matrix of the pre-trained code model; S4.4 combines the node embedding vectors of each node after word segmentation into a node embedding vector by averaging or weighted summation. S5. Graph Neural Network Construction: Using the edge relationships of the graph structure in step S3 and the embedding vectors in step S4 as inputs to the graph neural network, the structured representation of the code snippet is learned to obtain the constructed graph neural network; specifically including: S5.1 The edge relationships of the graph structure in step S3 and the embedding vector in step S4 are used as the input of the GNN; S5.2GNN reflects the position of each node in the graph structure and the characteristics of its neighbors by aggregating each node and its neighborhood information. S5.3 obtains the constructed graph neural network by stacking multiple GNN layers; S6. Code Model Parameter Fine-tuning: Using the constructed graph neural network as an adapter module, the pre-trained code model is trained using the training data from step S1. Efficient parameter fine-tuning is performed during training to obtain the fine-tuned code model; specifically including: S6.1 Load the pre-trained code model; S6.

2. Use the graph neural network constructed in step S5 as an adapter module and connect it to the pre-trained code model through soft hints or prefix fine-tuning. S6.3 On the training data of downstream tasks, only fine-tune the parameters of the adapter module in the code model, and freeze the rest of the code model; In step S6.2, the soft hint fine-tuning method is as follows: the vector output of the graph neural network is inserted into the input of the pre-trained code model as a "soft hint", and the "soft hint" is concatenated with the text embedding vector of the source code fragment as the input of the encoder in the code model; in the fine-tuning stage, only the parameters of the graph neural network vector are updated, and the rest of the code model is frozen. In step S6.2, the prefix fine-tuning is performed as follows: a learnable prefix vector is introduced into each multi-head attention layer of the encoder and decoder of the code model. The introduction method is to use the output of the graph embedding module as the prefix vector and concatenate it with the key value matrix of the multi-head attention layer.

2. The method for efficiently fine-tuning code models using abstract syntax trees according to claim 1, characterized in that, Step S1 includes: organizing the training data and test data into different data formats according to different downstream tasks, and cleaning and preprocessing the training data and test data, wherein the preprocessing includes: S1.1 Filter out invalid and duplicate code snippets; S1.2 Remove redundant information from the code snippet; S1.3 Format the code; The data formats for the different downstream tasks are specifically as follows: (1) Code summary task: <code snippet, corresponding natural language summary>; (2) Code translation task: <Source code, target language code>; (3) Vulnerability detection task: <code snippet, whether a specific vulnerability exists>.

3. The method for efficiently fine-tuning code models using abstract syntax trees according to claim 1, characterized in that, The process of automatically constructing the abstract syntax tree in step S2 includes: S2.1 Load the syntax file for a specific programming language; S2.

2. Use the syntax rules defined in the syntax file to parse code snippets and generate an abstract syntax tree; S2.3 Merge and delete redundant branch nodes and leaf nodes in the abstract syntax tree to obtain a simplified abstract syntax tree.

4. The method for efficiently fine-tuning code models using abstract syntax trees according to claim 1, characterized in that, In step S3, the method for transforming the abstract syntax tree into a graph structure is as follows: S3.

1. Use the root node of the abstract syntax tree as the starting node of the graph structure; S3.2 Traversing the Abstract Syntax Tree: S3.2.1 Map each node in the abstract syntax tree to a node in the graph structure and record the node type information; S3.2.2 Map the relationship between each node in the abstract syntax tree and its parent and child nodes as directed edges in a graph structure, as edge relationships; S3.

3. Prune the graph structure to remove redundant nodes and directed edges, generating the final graph structure.

5. An application apparatus for a code model obtained by the method of any one of claims 1-4 in a code task.

6. The application device according to claim 5, characterized in that, The code task is as follows: Code summary task: Input code snippets into the code model and decode them to generate corresponding natural language descriptions; Vulnerability detection task: Input code snippets into the code model, decode and generate vulnerability types or the probability of vulnerability existence; Code translation task: Input source code into a code model and decode it to generate code in the target language.

Images