Code abstract generation method and device

Abstract

The invention belongs to the technical field of code abstract generation, and particularly relates to a code abstract generation method and device.The code abstract generation method comprises the following specific steps that S1, function source codes and corresponding code abstracts are preprocessed, and a multi-modal function code abstract data set is obtained; s3, designing a two-stage fine tuning method oriented to a code abstract generation task, taking X0 and E as input, performing fine tuning on an original CodePTMs model to obtain a fine tuning prefix vector code abstract model CodePTMs' model, and outputting a code abstract; and S4, dividing the multimodal function code summary data set, and training the CodePTMs' model by using the multimodal function code summary data set. Through the two-stage fine tuning method oriented to the code abstract generation task, the deep semantic modeling capability of the model for the code can be improved, so that the accuracy of generating the code abstract by the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of code summary generation, and in particular to a code summary generation method and device. Background Art

[0002] Code summary generation refers to the task of automatically generating natural language text that can describe the function of a given function source code. When performing the code summary generation task, the code is often converted into an Abstract Syntax Tree (AST) as an intermediate representation. However, for more complex program fragments, the AST formed after parsing is usually large in scale and has a high structural depth. When such an AST is directly used for model training, it is easy for the context information of the AST to be lost during the transmission between nodes; moreover, the tree structure of the AST is sparse, making it difficult for the semantic relationship between nodes to be fully captured by the model, resulting in inefficient use of code structure information during training. It is difficult for the model to fully learn the grammatical and semantic features of the code, reducing the quality and accuracy of the generated summary.

[0003] Large-scale code pre-training models, CodePTMs, are specifically designed for processing programming language tasks. They accumulate rich understanding capabilities through massive amounts of code data and specific training tasks. In the following descriptions, large-scale code pre-training models, CodePTMs, will be referred to as CodePTMs models. For example, the CodeBert model is one such CodePTM. Existing CodePTMs models fail to specifically perform training tasks related to ASTs and are unable to understand ASTs, a tree-structured code representation. This limitation restricts CodePTMs models' ability to deeply model code semantics, making them particularly vulnerable when handling complex code scenarios.

[0004] Fine-tuning large models involves adjusting some parameters in a pre-trained model for a specific task or introducing a set of learnable parameters to improve the model's performance in a specific scenario. However, introducing prior knowledge of tree-structured code representations into a CodePTMs model that has not been trained on tree-structured code representation tasks through only one stage of fine-tuning can undermine the general knowledge already possessed by the CodePTMs model. In particular, when task data is limited, the model may overfit to the specific task data, resulting in unstable performance in other scenarios. Furthermore, one-stage fine-tuning can make it difficult to achieve a reasonable weight distribution between the tree-structured code modality and the code text sequence modality, resulting in poor information fusion between the two modalities and further reducing the model's overall performance.

[0005] Based on the above, the current method has the following shortcomings:

[0006] 1. The source code of the function used for code summary generation is usually complex, and the AST structure obtained by parsing it is complex. If directly used for model training, it is easy to cause the loss of node context information during training. In addition, the nodes on the AST are sparsely distributed, making it difficult for the model to effectively utilize node information.

[0007] 2. Existing CodePTMs models, lacking training on tree-structured code representations, are unable to understand such representations, resulting in poor performance in code summarization tasks.

[0008] 3. Introducing prior knowledge about the tree-structured code representation method by fine-tuning the CodePTMs model once may not only destroy the existing knowledge of the CodePTMs model, but also easily cause the model to overfit to specific tasks. It is also difficult to balance the weight distribution of the tree-structured code modality and the code text sequence modality representation, resulting in poor fusion of the two modal information and reduced model performance.

[0009] In summary, a method and device for generating a code summary are invented. Summary of the Invention

[0010] To solve the above technical problems, according to one aspect of the present invention, the present invention provides the following technical solutions:

[0011] A code summary generation method includes the following specific steps:

[0012] S1: Preprocess the function source code and its corresponding code summary to obtain a multimodal function code summary dataset

[0013] S2: For any triple Transform the FS-AST in the triple to obtain the first node feature embedding matrix and the adjacency matrix

[0014] S3: Design a two-stage fine-tuning method for code summary generation tasks. Taking X0 and E as input, fine-tune the original CodePTMs model to obtain the fine-tuned prefix vector code summary model CodePTMs' model, and output the code summary.

[0015] S4: Multimodal Function Code Summarization Dataset Divide and use it to train the CodePTMs' model;

[0016] S5: Apply the trained CodePTMs' model to the code summary generation task, and generate the corresponding code summary for the given function source code.

[0017] As a preferred solution of the code summary generation method of the present invention, the specific steps of S1 are as follows:

[0018] S11: Collect the public function source code and its corresponding code summary docstring, and use the abstract syntax tree parsing tool to parse the code into an abstract syntax tree AST;

[0019] S12: Perform post-order traversal on the AST to construct a complete syntax structure tree FS-AST;

[0020] S13: The source code of any function, the corresponding code summary docstring and the corresponding FS-AST generated after processing form a triple<code,docstring,FS-AST> , all triples constitute a multimodal function code summary dataset

[0021] As a preferred solution of the code summary generation method of the present invention, the specific steps of S12 are as follows:

[0022] S121: Use the Embedding method in the Pytorch framework to vectorize the text information of the node node and obtain the initial vector corresponding to the node The value of m is the embedding vector dimension size of the original CodePTMs model embedding layer;

[0023] S122: Calculate the hidden state of the node in Represents a 1×m-dimensional vector space:

[0024] When node is a leaf node, the calculation formula of h is:

[0025] h=W1γ

[0026] in, is the learnable parameter matrix in the hidden state encoder HSE, represents an m×m-dimensional vector space;

[0027] When node is a non-leaf node, the calculation formula of h is:

[0028]

[0029] Where tanh represents a nonlinear activation function; Ch(node) is the set of nodes in the subtree with node as the root; W2 is the learnable parameter matrix of RvNN in HSE; f∈Ch(node) is any node in Ch(node); h f represents the hidden state of f;

[0030] S123: If the type of the node node does not belong to the statement node type set Type, then replace node with h; otherwise, vectorize the subtree T with node as the root to form a subtree vector η, and then use η to replace the subtree T in the AST; after performing the above replacement operation on all nodes in the AST, a fully structured syntax tree FS-AST is obtained; where η is calculated as follows:

[0031] Perform the maximum pooling operation on the hidden state h of all nodes in T to form the subtree vector of T The calculation formula for η is:

[0032]

[0033] Among them, CH(T) represents the set of all nodes on T; g∈CH(T) represents any node in T; h g represents the hidden state of g; ∪ g∈CH(T) h g Represents a set consisting of the hidden states of all nodes in CH(T); the maxpooling operation represents selecting the maximum value along each dimension of all hidden states in the set to form a vector.

[0034] As a preferred solution of the code summary generation method of the present invention, the specific steps of S2 are as follows:

[0035] S21: Traverse FS-AST, for each current node c that is visited, embed the hidden state h or subtree vector η of c as the node feature to form a first node feature embedding matrix Where each node h or subtree vector η is a row of X0;

[0036] S22: Based on the connection relationship between each node in FS-AST, an adjacency matrix is ​​formed For any c i ,c j ∈FS-AST, the connection relationship between each node is stored in e i,j ∈E:

[0037]

[0038] As a preferred solution of the code summary generation method of the present invention, the specific steps of S3 are as follows:

[0039] S31: In the first fine-tuning stage, a prompt proxy module is constructed, X0 and E are input into the prompt proxy module to obtain a prompt embedding matrix Then concatenate X0 and X1 to obtain the fused embedding matrix X2;

[0040] S32: In the second fine-tuning stage, the parameters of the original CodePTMs model are expanded using the prefix vector ΔW to obtain a fine-tuned prefix vector code summary model CodePTMs' model;

[0041] S33: Input X2 into the CodePTMs' model and pass it through L layers of Transformer Layers to extract semantic features and obtain the summary representation H L ; Then H L Input to the prediction head to generate a code summary; where H L Calculated by the following iterative equation:

[0042] H 0 =X2

[0043] H l =TransformerLayer(H l-1 ),1≤l≤L

[0044] Among them, H l Represents the output of the lth layer Transformer Layer, H l-1 Represents the output of the l-1th layer TransformerLayer, and TransformerLayer represents the calculation performed in the Transformer Layer.

[0045] As a preferred solution of the code summary generation method of the present invention, the specific steps of S31 are as follows:

[0046] S311: Construct a prompt agent module, input the first node feature embedding matrix X0 and the adjacency matrix E into the prompt agent module to obtain the prompt embedding matrix

[0047] S312: Concatenate X0 and X1 to obtain the fused embedding matrix X2:

[0048] X2=Concat(X1,X0)

[0049] Among them, X2 contains the text information of each node on FS-AST and the structural information between nodes;

[0050] The specific steps of S311 are as follows:

[0051] S3111: Input X0 and E into the graph attention encoder to obtain the second node feature embedding matrix

[0052] S3112: Perform flatten operation on X' to obtain the first feature representation vector

[0053] β1=flatten(X')

[0054] in, represents a (n×m)×1-dimensional vector space;

[0055] S3113: Input β1 into the first multi-layer perceptron MLP1 for calculation to obtain the third feature representation vector

[0056] S3114: Reshape β3 to obtain the hint embedding matrix

[0057] X1=reshape(β3,p,m)

[0058] in, Representing a p×m-dimensional vector space, the calculated X1 also contains the FS-AST structural semantics, that is, the structural information between nodes on the FS-AST;

[0059] The specific steps of S3111 are as follows:

[0060] S31111: Map X0 to the query space, key space, and value space respectively to obtain the query vector matrix Q, key vector matrix K, and value vector matrix V. The calculation formulas are as follows:

[0061] Q=X0W Q1 , K=X0W K1 , V=X0W V1

[0062] Among them, the query matrix Bond Matrix Value Matrix Both are the learnable attention mechanism projection matrices in GAT Coder;

[0063] S31112: Calculate the attention score matrix between all nodes that make up X0

[0064]

[0065] S31113: Use -∞ to mask the attention scores between non-adjacent nodes in S to obtain the masked attention score matrix S'. For any s' a,r ∈S', we have:

[0066]

[0067] Where node a,r∈D(X0), D(X0) represents the set of all node features that constitute X0 embedded in the corresponding nodes; s a,r ∈S represents the corresponding scores of a and r in S; s' a,r ∈S' represents the corresponding scores of a and r in S'; edge e a,r ∈E,e a,r =1 means a and r are adjacent, e a,r =0 means a and r are not adjacent, -∞ means a very small value;

[0068] S31114: Perform softmax normalization on S' to obtain the attention coefficient matrix α is calculated as follows:

[0069] α=softmax(S')

[0070] S31115: Combine the value vector matrix V and calculate the second node feature embedding matrix X':

[0071] X'=αV

[0072] Among them, the calculated X' contains the FS-AST structural semantics, that is, the structural information between nodes on the FS-AST;

[0073] The specific steps of S3113 are as follows:

[0074] S31131: Input β1 into the first layer of the first multi-layer perceptron MLP1, perform linear transformation and activate it using the ReLU activation function to obtain the second feature representation vector

[0075] β2=ReLU(W4β1+b4)

[0076] S31132: Input β2 into the second layer of the first multi-layer perceptron MLP1, perform linear transformation and activate it using the same ReLU activation function as the first layer to obtain the third feature representation vector

[0077] β3=ReLU(W5β2+b5)

[0078] in, Both are learnable parameter matrices; All are bias terms; represents a (n×m)×(n×m)-dimensional vector space, represents a (p×m)×(n×m)-dimensional vector space, represents a (n×m)×1-dimensional vector space, Represents a (p×m)×1-dimensional vector space.

[0079] As a preferred solution of the code summary generation method of the present invention, the specific steps of S32 are as follows:

[0080] S321: Calculate ΔW using low-rank matrices A and B:

[0081] ΔW=AB

[0082] The low-rank matrix A is an m×o-dimensional learnable parameter matrix; the low-rank matrix B is an o×Δu-dimensional learnable parameter matrix. Compared with directly updating ΔW, calculating ΔW by updating the low-rank matrices A and B can improve the generalization ability and training stability of the CodePTMs' model.

[0083] S322: Compare ΔW with the query matrix W in each layer of Transformer Layer Q2 , bond matrix W K2 Splicing to expand W Q2 and W K2 The dimension of is , and a fine-tuned prefix vector code summary model CodePTMs' model is obtained; the calculation formula is as follows:

[0084] W Q2 =Concat(ΔW,W Q2 ,axis=1)

[0085] W K2 =Concat(ΔW,W K2 ,axis=1)

[0086] Concat is a concatenation operation, and the parameter axis=1 specifies that the concatenation operation is performed along the horizontal direction of the matrix.

[0087] As a preferred solution of the code summary generation method of the present invention, the specific steps of S4 are as follows:

[0088] S41: Divide into training data set, test data set and validation data set;

[0089] S42: Randomly initialize the learnable parameter matrix set TD; and randomly initialize the low-rank matrices A and B in the CodePTMs' model; where TD={W Q1 ,W K1 ,W V1 ,W4,W5,b4,b5} is the set of learnable parameter matrices in the prompt agent module;

[0090] S43: During the first fine-tuning phase, all learnable parameter matrices in TD are updated by performing edge prediction on FS-AST. The edge prediction task on FS-AST involves using the obtained hint embedding matrix X1 to determine whether there is a direct connection between any two nodes on FS-AST.

[0091] S44: In the second fine-tuning phase, the hyperparameters set in the first fine-tuning phase are used. A suitable loss function and optimization algorithm are selected to update the parameters of the low-rank matrices A and B. The performance of the prefix vector ΔW is continuously evaluated during training, and the hyperparameters are dynamically adjusted during training to obtain a trained CodePTMs' model.

[0092] As a preferred solution of the code summary generation method of the present invention, the specific steps of S43 are as follows:

[0093] S431: For any node q, w∈D(X0), and Concatenate with the prompt embedding matrix X1 and then perform flatten operation to obtain the first structure representation vector

[0094]

[0095] Among them, D(X0) represents the set of all node features that constitute X0 embedded in the corresponding nodes, represents the node feature embedding corresponding to node q in X0, represents the node feature embedding corresponding to node w in X0, represents a vector space of ((p+2)×m)×1 dimensions;

[0096] S432: Input z1 into the second multi-layer perceptron MLP2 to obtain the classification result

[0097] S433: Set the hyperparameters required for training, including batch size, training rounds, gradient clipping threshold, learning rate, and weight decay coefficient. Then, select an appropriate loss function and optimization algorithm to update the parameters of all learnable parameter matrices in TD. During training, continuously evaluate the performance of the prompt agent module and the second multi-layer perceptron MLP2 to facilitate dynamic adjustment of hyperparameters and improve training efficiency. Finally, a trained prompt agent module is obtained.

[0098] The specific steps of S432 are as follows:

[0099] S4321: Input z1 into the first layer of the second multi-layer perceptron MLP2, perform linear transformation and activate it using the ReLU activation function to obtain the second structure representation vector

[0100] z2=ReLU(W6z1+b6)

[0101] S4322: Input z2 into the second layer of the second multi-layer perceptron MLP2, perform linear transformation and apply softmax to normalize the data to obtain the classification result.

[0102]

[0103] in, are all learnable parameter matrices, Both are bias terms. The structure of the second multi-layer perceptron MLP2 is exactly the same as that of the first multi-layer perceptron MLP1. The function of the second multi-layer perceptron MLP2 is to evaluate the ability of X1 to contain the structural information between the nodes on FS-AST. represents a 2×1 dimensional vector space, represents an m×1-dimensional vector space, represents an m×((p+2)×m)-dimensional vector space, Represents a 2×m-dimensional vector space.

[0104] A code summary generating device, comprising:

[0105] The preprocessing module is used to preprocess the function source code and its corresponding code summary to obtain a multimodal function code summary dataset

[0106] Conversion module for any triple Transform the FS-AST in the triple to obtain the first node feature embedding matrix and the adjacency matrix

[0107] A two-stage fine-tuning design module is used to design a two-stage fine-tuning method for code summary generation tasks. It takes X0 and E as input, fine-tunes the original CodePTMs model, obtains the fine-tuned prefix vector code summary model CodePTMs' model, and outputs the code summary.

[0108] Partitioning module: for multimodal function code summary dataset Divide and use it to train the CodePTMs' model;

[0109] The application module is used to apply the trained CodePTMs' model to the code summary generation task and generate the corresponding code summary for the given function source code.

[0110] Compared with existing technologies:

[0111] The present invention constructs a simple FS-AST through AST, which can avoid directly training the complex AST, resulting in the problem of loss of context information under the node during training. Through a two-stage fine-tuning method for code summary generation tasks, in the first fine-tuning stage, FS-AST structural semantic-related knowledge is introduced into the original CodePTMs model. In the second fine-tuning stage, the weight distribution of the original CodePTMs model for the FS-AST modality and code text sequence modality representation is balanced, which can enhance the model's deep semantic modeling ability for code, thereby improving the accuracy of the model in generating code summaries. In addition, the present invention can also adapt to different types of code and code-related downstream tasks, thereby improving versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 It is a schematic diagram of the process of the present invention;

[0113] Figure 2 This is a flow chart of the two-stage fine-tuning method of the present invention;

[0114] Figure 3 This is a diagram showing the structure of the agent module of the present invention;

[0115] Figure 4 This is the original CodePTMs model structure diagram of the present invention;

[0116] Figure 5 This is the original CodeBert model structure diagram of the present invention;

[0117] Figure 6 This is the structure diagram of the CodeBert' model of the present invention. DETAILED DESCRIPTION

[0118] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0119] The present invention provides a code summary generation method, please refer to Figures 1-6 , including the following specific steps:

[0120] S1: Preprocess the function source code and its corresponding code summary to obtain a multimodal function code summary dataset

[0121] First, collect the public function source code and its corresponding code summary. For each set of function source code and its corresponding code summary docstring, convert the code into an abstract syntax tree AST, and then construct a complete syntax tree FS-AST, and finally form a triple<code,docstring,FS-AST> , all triples constitute a multimodal function code summary dataset

[0122] The specific steps of S1 are as follows:

[0123] S11: Collect the public function source code and its corresponding code summary docstring, and use the abstract syntax tree parsing tool to parse the code into an abstract syntax tree AST.

[0124] in:

[0125] (1) In the implementation process of the present invention, public function source code summary datasets are downloaded from Github, Kaggle and other online channels, such as CodeSearchNet, CodeXGLUE, Funcom, etc. Each dataset contains several public function source code codes and their corresponding code summary docstrings.

[0126] (2) Abstract syntax tree parsing tools: These tools are used to generate abstract syntax trees for function source code. They are often used in scenarios such as code static analysis, refactoring, optimization, and security testing. Common tools of this type include: 1) Tree-sitter, an efficient parsing library written in C that supports abstract syntax tree generation for multiple languages; 2) javalang, written in Python, which only supports abstract syntax tree generation for Java code.

[0127] (3) Abstract Syntax Tree (AST): A tree-structured code representation where nodes represent grammatical units in the code, and the relationships between nodes reflect the grammatical structure of the code. All nodes in the AST are directly parsed from the source code and are called code nodes.

[0128] S12: Perform post-order traversal on the AST to construct a complete syntax structure tree FS-AST.

[0129] In the process of post-order traversal of AST, the initial vector of the currently visited node is first calculated. Then calculate the hidden state of the node If the type of node does not belong to the statement node type set Type, then replace node with h; otherwise, vectorize the subtree T with node as the root to form a subtree vector Finally, replace T with η. When all nodes in the AST are visited, a fully structured syntax tree FS-AST is obtained.

[0130] Among them, the statement node type set Type = {IfStatement, WhileStatement, ForStatement}, IfStatement represents the if conditional judgment statement for executing process branch selection; WhileStatement represents the while loop structure statement that repeatedly executes a certain section of code when specific conditions are met; ForStatement represents the for counting loop statement used for iterative control and has initialization, conditional judgment and iteration process.

[0131] Beneficial results of S12: When the AST is large, the nodes in the AST are sparsely distributed and difficult to use effectively. By replacing nodes or subtrees in the AST and constructing an FS-AST, we can preserve the AST's node text information and the structural information between nodes as much as possible while reducing its complexity, making it easier for the model to capture effective features.

[0132] The specific steps of S12 are as follows:

[0133] S121: Use the Embedding method in the Pytorch framework to vectorize the text information of the node node and obtain the initial vector corresponding to the node The value of m is the embedding vector dimension size of the original CodePTMs model embedding layer;

[0134] in:

[0135] (1) Embedding method: The Pytorch framework provides an implementation of this method, which uses embedding vectors to map discrete input sequences into a continuous, low-dimensional vector space. These embedding vectors are learned through training and can capture the semantic or category relationship between inputs.

[0136] (2) Embedding layer: An important component of the original CodePTMs model for processing discrete symbolic inputs. It consists of a series of embedding vectors, whose main function is to map discrete input sequences (such as words, subwords, identifiers, etc.) into low-dimensional, continuous vector representations. Compared with high-dimensional sparse one-hot encoding, the embedding layer can capture the semantic and structural relationships between inputs in a lower-dimensional space. The embedding vectors are continuously optimized through backpropagation during training, thereby providing the model with more expressive input features. It is a key bridge connecting the symbolic world and the semantic space of neural networks.

[0137] S122: Calculate the hidden state of the node in Represents a 1×m-dimensional vector space:

[0138] When node is a leaf node, the calculation formula of h is:

[0139] h=W1γ

[0140] in is the learnable parameter matrix in the hidden state encoder HSE, represents an m×m-dimensional vector space;

[0141] Among them, the hidden state encoder (HSE) is composed of a learnable parameter matrix W1 and a recursive neural network (RvNN), which is used to calculate the hidden state h of each node in the tree.

[0142] When node is a non-leaf node, the calculation formula of h is:

[0143]

[0144] Where tanh represents a nonlinear activation function; Ch(node) is the set of nodes in the subtree with node as the root; W2 is the learnable parameter matrix of RvNN in HSE; f∈Ch(node) is any node in Ch(node); h f represents the hidden state of f;

[0145] Among them, tanh is a commonly used nonlinear activation function that can compress the input to the interval of (-1,1). Tanh has the characteristic of zero center, which is conducive to accelerating the convergence speed of neural networks. Therefore, it is often used in hidden layers to enhance the expressive ability of the model.

[0146] S123: If the type of the node node does not belong to the statement node type set Type, then node is replaced with h; otherwise, the subtree T with node as the root is vectorized to form a subtree vector η, and then η is used to replace the subtree T in the AST; after performing the above replacement operation on all nodes on the AST, a fully structured syntax tree FS-AST is obtained; FS-AST refers to a fully structured syntax tree formed after the nodes or subtrees on the AST are vectorized; where η is calculated as follows:

[0147] Perform the maximum pooling operation on the hidden state h of all nodes in T to form the subtree vector of T The calculation formula for η is:

[0148]

[0149] Among them, CH(T) represents the set of all nodes on T; g∈CH(T) represents any node in T; h g represents the hidden state of g; U g∈CH(T) h g Represents a set consisting of the hidden states of all nodes in CH(T); the maxpooling operation represents selecting the maximum value along each dimension of all hidden states in the set to form a vector;

[0150] Maxpooling is a commonly used operation to extract the maximum value of a local region from a set of input data. It slides a sliding window across the two-dimensional or multi-dimensional space of the input data, selecting the maximum value in each window as the representative region. This operation can extract the most important information from a set of data while reducing the data's dimensionality, accelerating subsequent computations, and improving the robustness of the model.

[0151] S13: The source code of any function, the corresponding code summary docstring and the corresponding FS-AST generated after processing form a triple<code,docstring,FS-AST> , all triples constitute a multimodal function code summary dataset

[0152] S2: For any triple Transform the FS-AST in the triple to obtain the first node feature embedding matrix and the adjacency matrix

[0153] The specific steps of S2 are as follows:

[0154] S21: Traverse FS-AST, for each current node c that is visited, embed the hidden state h or subtree vector η of c as the node feature to form a first node feature embedding matrix Where each node h or subtree vector η is a row of X0;

[0155] Where n represents the total number of nodes on the FS-AST, m is the embedding vector dimension of the original CodePTMs model embedding layer, and the FS-AST traversal method can be any of the pre-order traversal, in-order traversal, and post-order traversal methods. represents an n×m-dimensional vector space, Represents an n×n dimensional vector space.

[0156] S22: Based on the connection relationship between each node in FS-AST, an adjacency matrix is ​​formed For any c i ,c j ∈FS-AST, the connection relationship between each node is stored in e i,j ∈E:

[0157]

[0158] S3: Design a two-stage fine-tuning method for the code summary generation task. Taking X0 and E as input, fine-tune the original CodePTMs model to obtain the fine-tuned prefix vector code summary model CodePTMs' model, and output the code summary.

[0159] In the two-stage fine-tuning method for code summary generation tasks, the original CodePTMs model is fine-tuned in two stages: in the first fine-tuning stage, a prompt proxy module is constructed, X0 and E are input into the prompt proxy module to generate the prompt embedding matrix Then concatenate X0 and X1 to form the fused embedding matrix In the second fine-tuning stage, by introducing the prefix vector Expand the parameters of the original CodePTMs model to obtain the fine-tuned prefix vector code summary model CodePTMs' model. Finally, input X2 into the CodePTMs' model to generate the code summary.

[0160] Beneficial results of S3: The original CodePTMs model, which has not been pre-trained on FS-AST-related tasks, cannot understand the tree-structured code representation of FS-AST. To address this issue, this method first forms a fused embedding matrix X2 in the first fine-tuning stage, introducing semantic knowledge about the FS-AST structure into the original CodePTMs model. Then, in the second fine-tuning stage, the original CodePTMs model is fine-tuned on the prefix vector using the code summarization task, further improving the model's performance on the code summarization task.

[0161] The specific steps of S3 are as follows:

[0162] S31: In the first fine-tuning stage, a prompt proxy module is constructed, X0 and E are input into the prompt proxy module to obtain a prompt embedding matrix Then concatenate X0 and X1 (i.e. Figure 2 The concatenation operation Concat in is used to obtain the fused embedding matrix X2.

[0163] The specific steps of S31 are as follows:

[0164] S311: Construct a prompt agent module, whose structure is as follows Figure 3 As shown, the first node feature embedding matrix X0 and the adjacency matrix E are input into the above prompt agent module to obtain the prompt embedding matrix

[0165] The Graph Attention Network Coder (GAT Coder) is a publicly available encoder that uses an attention mechanism to encode graph-structured data. By introducing an attention mechanism, the encoder achieves adaptive weighted aggregation of neighbor node information, thereby improving its expressiveness when processing unstructured and heterogeneous graph data. In the attention mechanism, the input features of each node are first mapped to a query space (Query), a key space (Key), and a value space (Value) through a set of learnable attention mechanism projection matrices, constructing a query vector, a key vector, and a value vector, respectively. The query space describes the information requirements of the central node, the key space describes the index features of the neighbor nodes, and the value space represents the actual information that can be aggregated. Attention weights are generated by calculating the correlation between the query vector and the key vector of the corresponding neighbor, and are normalized using softmax. Finally, the attention weights are used to weight the value vectors of the neighbor nodes to update the node features.

[0166] A fully connected neural network (FCNN) is a neural network consisting of multiple fully connected layers, where neurons in each layer are connected to all neurons in the previous layer. It is commonly used for feature extraction and pattern recognition. It processes information through layer-by-layer calculations across input, hidden, and output layers, and is trained using backpropagation and gradient descent algorithms. It is widely used in fields such as image recognition, speech processing, and predictive modeling.

[0167] The specific steps of S311 are as follows:

[0168] S3111: Input X0 and E into the graph attention encoder to obtain the second node feature embedding matrix

[0169] The specific steps of S3111 are as follows:

[0170] S31111: Map X0 to the query space, key space, and value space respectively to obtain the query vector matrix Q, key vector matrix K, and value vector matrix V. The calculation formulas are as follows:

[0171] Q=X0W Q1 , K=X0W K1 , V=X0W V1

[0172] Among them, the query matrix Bond Matrix Value Matrix Both are the learnable attention mechanism projection matrices in GAT Coder;

[0173] S31112: Calculate the attention score matrix between all nodes that make up X0

[0174]

[0175] Among them, LeakyReLU is an improved ReLU (Rectified Linear Unit) activation function. It introduces a small negative slope parameter based on the classic ReLU function, which can handle the situation where the input is less than 0 and alleviate the "dead neurons problem" (Dead Neurons) that may be caused by the ReLU function.

[0176] S31113: Use -∞ to mask the attention scores between non-adjacent nodes in S to obtain the masked attention score matrix S'. For any s' a,r ∈S', we have:

[0177]

[0178] Where node a,r∈D(X0), D(X0) represents the set of all node features that constitute X0 embedded in the corresponding nodes; s a,r ∈S represents the corresponding scores of a and r in S; s' a,r ∈S' represents the corresponding scores of a and r in S'; edge e a,r ∈E,e a,r =1 means a and r are adjacent, e a,r =0 means a and r are not adjacent, -∞ means a very small value;

[0179] Example: In actual engineering application practice, the value of -∞ can be 1.0×2 -126 .

[0180] S31114: Perform softmax normalization on S' to obtain the attention coefficient matrix α is calculated as follows:

[0181] α=softmax(S')

[0182] Softmax is a normalization operation commonly used in classification tasks. Its main function is to convert a set of real numbers into a probability distribution. It takes the exponential of each input value and divides it by the sum of all the input exponentials, so that the output is between (0, 1) and the sum is 1. This can intuitively represent the probability of each category being selected.

[0183] S31115: Combine the value vector matrix V and calculate the second node feature embedding matrix X':

[0184] X'=αV

[0185] The calculated X' contains the FS-AST structural semantics, that is, the structural information between nodes on the FS-AST.

[0186] S3112: Perform flatten operation on X' to obtain the first feature representation vector

[0187] β1=flatten(X')

[0188] in, represents a (n×m)×1-dimensional vector space;

[0189] Among them, the flatten operation is an operation used to convert a multidimensional array into a one-dimensional array. Specifically, it extracts the elements in the original data structure in sequence and puts them into a one-dimensional structure.

[0190] S3113: Input β1 into the first multi-layer perceptron MLP1 for calculation to obtain the third feature representation vector

[0191] Among them, the first multilayer perceptron MLP1 is a multilayer perceptron composed of two layers of fully connected neural network (FCNN), in which the first layer contains a learnable parameter matrix W4 and a bias b4, which is used to map the input to the latent space, extract features and introduce nonlinear expression capabilities; the second layer contains a learnable parameter matrix W5 and a bias b5, which is used to output the results.

[0192] The specific steps of S3113 are as follows:

[0193] S31131: Input β1 into the first layer of the first multi-layer perceptron MLP1, perform linear transformation and activate it using the ReLU activation function to obtain the second feature representation vector

[0194] β2=ReLU(W4β1+b4)

[0195] S31132: Input β2 into the second layer of the first multi-layer perceptron MLP1, perform linear transformation and activate it using the same ReLU activation function as the first layer to obtain the third feature representation vector

[0196] β3=ReLU(W5β2+b5)

[0197] in, All are learnable parameter matrices; All are bias terms; represents a (n×m)×(n×m)-dimensional vector space, represents a (p×m)×(n×m)-dimensional vector space, represents a (n×m)×1-dimensional vector space, Represents a (p×m)×1-dimensional vector space.

[0198] ReLU is a widely used nonlinear activation function in deep learning. It outputs zero for negative inputs and maintains a linear relationship with positive inputs. ReLU is computationally simple, effectively mitigates the vanishing gradient problem, and accelerates neural network training, making it the mainstream activation function for hidden layers in modern neural networks.

[0199] S3114: Reshape β3 to obtain the hint embedding matrix

[0200] X1=reshape(β3,p,m)

[0201] in, Representing a p×m-dimensional vector space, the calculated X1 also contains the FS-AST structural semantics, that is, the structural information between nodes on FS-AST.

[0202] The reshape operation is an operation used to transform data from one shape to another.

[0203] S312: Concatenate X0 and X1 to obtain the fused embedding matrix X2:

[0204] X2=Concat(X1,X0)

[0205] Among them, X2 contains the text information of each node on FS-AST and the structural information between nodes.

[0206] S32: In the second fine-tuning stage, the parameters of the original CodePTMs model are expanded using the prefix vector ΔW to obtain a fine-tuned prefix vector code summary model CodePTMs' model.

[0207] The prefix vector ΔW is a vector of m×Δu dimensions; the original CodePTMs model refers to a large code pre-training model that maintains the original structure and parameter settings and has not undergone any downstream task adaptation or structural modification. The structure of the original CodePTMs model is as follows Figure 4 As shown in the figure, there are L layers of Transformer Layer stacked together. Each layer of Transformer Layer applies the multi-head attention mechanism (Multi-Head Attention). After the Lth layer of Transformer Layer, a prediction head is connected to generate code summaries.

[0208] The specific steps of S32 are as follows:

[0209] S321: Calculate ΔW using low-rank matrices A and B:

[0210] ΔW=AB

[0211] The low-rank matrix A is an m×o-dimensional learnable parameter matrix; the low-rank matrix B is an o×Δu-dimensional learnable parameter matrix. Compared with directly updating ΔW, calculating ΔW by updating the low-rank matrices A and B can improve the generalization ability and training stability of the CodePTMs' model.

[0212] S322: Compare ΔW with the query matrix W in each layer of Transformer Layer Q2 , bond matrix W K2 Splicing to expand W Q2 and W K2 The dimension of is , and a fine-tuned prefix vector code summary model CodePTMs' model is obtained; the calculation formula is as follows:

[0213] W Q2 =Concat(ΔW,W Q2 ,axis=1)

[0214] W K2 =Concat(ΔW,W K2 ,axis=1)

[0215] Concat is a concatenation operation, and the parameter axis=1 specifies that the concatenation operation is performed along the horizontal direction of the matrix.

[0216] S33: Input X2 into the CodePTMs' model and pass it through L layers of Transformer Layers to extract semantic features and obtain the summary representation H L ; Then H L Input to the prediction head to generate a code summary; where H L Calculated by the following iterative equation:

[0217] H 0 =X2

[0218] H l =TransformerLayer(H l-1 ),1≤l≤L

[0219] Among them, H l Represents the output of the lth layer Transformer Layer, H l-1 Represents the output of the l-1th layer TransformerLayer, and TransformerLayer represents the calculation performed in the Transformer Layer.

[0220] Example:

[0221] The CodeBert model is a specific CodePTMs model. In this embodiment, the prefix vector ΔW is used to fine-tune the original CodeBert model.

[0222] The CodeBert model that has not been fine-tuned is called the original CodeBert model. The original CodeBert model is a large code pre-trained model that maintains the original structure and parameter settings and has not undergone any downstream task adaptation or structural modification. It is composed of 12 layers of Transformer Layer stacking. Each Transformer Layer has a standard encoder structure formed by a multi-head attention mechanism (Multi-Head Attention), a feedforward neural network (Feed Forward), and a residual connection and layer normalization (Add&LayerNorm) module. The structure of the original CodePTMs model is as follows Figure 5 As shown, the parameters of the original CodeBert model are expanded using the prefix vector ΔW to obtain the CodeBert' model. The structure of the CodeBert' model is as follows Figure 6 shown.

[0223] X2 is input into the CodeBert' model and passes through 12 layers of Transformer Layer in sequence to extract semantic features and obtain the summary representation H 12 ; Then H 12 Input to the prediction head to generate a code summary. 12 Calculated by the following iterative equation:

[0224] H 0 =X2

[0225]

[0226] in, It is the specific TransformerLayer calculation method in the CodeBert' model, H l Represents the output of the lth layer Transformer Layer, H l-1 Represents the output of the l-1th layer Transformer Layer, LN represents the layer normalization calculation (i.e. Figure 6 LayerNorm in Transformer Layer), MH represents the multi-head attention mechanism in Transformer Layer (i.e. Figure 6 The calculation of Multi-HeadAttention in Transformer Layer is performed by FFN, which means the feedforward neural network in Transformer Layer (i.e. Figure 6 The calculation of FeedForward in Indicates residual connection (i.e. Figure 6 in the Add).

[0227] S4: Multimodal Function Code Summarization Dataset Divide and use it to train the CodePTMs' model.

[0228] The specific steps of S4 are as follows:

[0229] S41: Divide into training dataset, test dataset and validation dataset.

[0230] S42: Randomly initialize the learnable parameter matrix set TD; and randomly initialize the low-rank matrices A and B in the CodePTMs' model; where TD={W Q1 ,W K1 ,W V1 ,W4,W5,b4,b5} is the set of learnable parameter matrices in the prompt agent module.

[0231] S43: During the first fine-tuning phase of training, all learnable parameter matrices in TD are updated by performing edge prediction on FS-AST. The edge prediction task on FS-AST involves using the obtained hint embedding matrix X1 to determine whether there is a direct connection between any two nodes on FS-AST.

[0232] The specific steps of S43 are as follows:

[0233] S431: For any node q, w∈D(X0), and Concatenate with the prompt embedding matrix X1 and then perform flatten operation to obtain the first structure representation vector

[0234]

[0235] Among them, D(X0) represents the set of all node features that constitute X0 embedded in the corresponding nodes, represents the node feature embedding corresponding to node q in X0, represents the node feature embedding corresponding to node w in X0, Represents a ((p+2)×m)×1-dimensional vector space.

[0236] S432: Input z1 into the second multi-layer perceptron MLP2 to obtain the classification result

[0237] The specific steps of S432 are as follows:

[0238] S4321: Input z1 into the first layer of the second multi-layer perceptron MLP2, perform linear transformation and activate it using the ReLU activation function to obtain the second structure representation vector

[0239] z2=ReLU(W6z1+b6)

[0240] S4322: Input z2 into the second layer of the second multi-layer perceptron MLP2, perform linear transformation and apply softmax to normalize the data to obtain the classification result.

[0241]

[0242] in, are all learnable parameter matrices, Both are bias terms. The structure of the second multi-layer perceptron MLP2 is exactly the same as that of the first multi-layer perceptron MLP1. The function of the second multi-layer perceptron MLP2 is to evaluate the ability of X1 to contain the structural information between the nodes on FS-AST. represents a 2×1 dimensional vector space, represents an m×1-dimensional vector space, represents an m×((p+2)×m)-dimensional vector space, Represents a 2×m-dimensional vector space.

[0243] In this patent solution, the first multi-layer perceptron MLP1 is used to perform further nonlinear transformation on the second node feature embedding matrix X' having the structural information between nodes, mapping X' to a dimension of a specified size to form a prompt embedding matrix X1 having the FS-AST structural semantics. The second multi-layer perceptron MLP2 is used during the training process of the first fine-tuning stage to evaluate the ability of X1 to contain the structural information between nodes on the FS-AST.

[0244] S433: Set the hyperparameters required for training, including batch size, training rounds, gradient clipping threshold, learning rate, and weight decay coefficient. Then select a suitable loss function and optimization algorithm to update the parameters of all learnable parameter matrices in TD. During the training process, continuously evaluate the performance of the prompt agent module and the second multi-layer perceptron MLP2 to facilitate dynamic adjustment of hyperparameters and improve training efficiency. Finally, a trained prompt agent module is obtained.

[0245] S44: In the second fine-tuning phase, the hyperparameters set in the first fine-tuning phase are used. A suitable loss function and optimization algorithm are selected to update the parameters of the low-rank matrices A and B. The performance of the prefix vector ΔW is continuously evaluated during training, and the hyperparameters are dynamically adjusted during training to obtain a trained CodePTMs' model.

[0246] Example: It is divided into a training data set, a test data set, and a validation data set in a ratio of 8:1:1. In the actual implementation of the solution of the present invention, the batch size is set to 32, the training round is 100, the gradient clipping threshold is 2.5, the learning rate is 1e-3, and the weight decay coefficient is 1e-5; in the first fine-tuning training stage, the loss function selected is the binary cross entropy loss function, which is a commonly used loss function for binary classification tasks and is used to measure the gap between the model prediction result and the true label; the optimization algorithm selected is the Adam optimization algorithm, which is an optimization method that combines adaptive learning rate and momentum mechanism. It can dynamically adjust the learning rate of each parameter during the training process, thereby accelerating convergence and improving training efficiency. In the second fine-tuning training stage, the loss function selected is the cross entropy loss function, which is a commonly used loss function for multi-category classification or sequence generation tasks. By calculating the difference between the probability distribution predicted by the model and the true label distribution, the prediction accuracy of the model when generating each target word is measured; the optimization algorithm selected is also the Adam optimization algorithm.

[0247] The specific formula of the binary cross entropy loss function is:

[0248]

[0249] Among them, BS represents the total number of samples used in one training, y bs Represents the true label value of the bsth sample, Represents the prediction result of the bsth sample. Log is a logarithmic function used to amplify the deviation between the prediction and the true value.

[0250] The specific formula of the cross entropy loss function is:

[0251]

[0252] Among them, BC represents the total number of tokens predicted in this training, and VS represents the total number of predicted categories, i.e. the size of the vocabulary. Represents the true label value of the bc-th sample in the vs-th class, Represents the predicted probability distribution that the bcth sample belongs to the vsth class. The logarithmic function is used to amplify the deviation between the prediction and the true value.

[0253] S5: Apply the trained CodePTMs' model to the code summary generation task, and generate the corresponding code summary for the given function source code.

[0254] A code summary generating device, comprising:

[0255] The preprocessing module is used to preprocess the function source code and its corresponding code summary to obtain a multimodal function code summary dataset

[0256] Conversion module for any triple Transform the FS-AST in the triple to obtain the first node feature embedding matrix and the adjacency matrix

[0257] A two-stage fine-tuning design module is used to design a two-stage fine-tuning method for code summary generation tasks. It takes X0 and E as input, fine-tunes the original CodePTMs model, obtains the fine-tuned prefix vector code summary model CodePTMs' model, and outputs the code summary.

[0258] Partitioning module: for multimodal function code summary dataset Divide and use it to train the CodePTMs' model;

[0259] The application module is used to apply the trained CodePTMs' model to the code summary generation task and generate the corresponding code summary for the given function source code.

[0260] Although the present invention has been described above with reference to embodiments, various modifications may be made thereto and equivalent components may be substituted without departing from the scope of the present invention. In particular, as long as there are no structural conflicts, the various features of the embodiments disclosed herein may be combined with each other in any manner, and the omission of an exhaustive description of such combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A code summary generation method, characterized in that: The specific steps are as follows: S1: Preprocess the function source code and its corresponding code summary to obtain a multimodal function code summary dataset S2: For any triple Transform the FS-AST in the triple to obtain the first node feature embedding matrix and the adjacency matrix S3: Design a two-stage fine-tuning method for code summary generation tasks. Taking X0 and E as input, fine-tune the original CodePTMs model to obtain the fine-tuned prefix vector code summary model CodePTMs' model, and output the code summary. S4: Multimodal Function Code Summarization Dataset Divide and use it to train the CodePTMs' model; S5: Apply the trained CodePTMs' model to the code summary generation task, and generate the corresponding code summary for the given function source code.

2. A code summary generation method according to claim 1, characterized in that: The specific steps of S1 are as follows: S11: Collect the public function source code and its corresponding code summary docstring, and use the abstract syntax tree parsing tool to parse the code into an abstract syntax tree AST; S12: Perform post-order traversal on the AST to construct a complete syntax structure tree FS-AST; S13: The source code of any function, the corresponding code summary docstring and the corresponding FS-AST generated after processing form a triple<code,docstring,FS-AST> , all triples constitute a multimodal function code summary dataset 3. A code summary generation method according to claim 2, characterized in that: The specific steps of S12 are as follows: S121: Use the Embedding method in the Pytorch framework to vectorize the text information of the node node and obtain the initial vector corresponding to the node The value of m is the embedding vector dimension size of the original CodePTMs model embedding layer; S122: Calculate the hidden state of the node in Represents a 1×m-dimensional vector space: When node is a leaf node, the calculation formula of h is: h=W1γ in is the learnable parameter matrix in the hidden state encoder HSE, represents an m×m-dimensional vector space; When node is a non-leaf node, the calculation formula of h is: Where tanh represents a nonlinear activation function; Ch(node) is the set of nodes in the subtree with node as the root; W2 is the learnable parameter matrix of RvNN in HSE; f∈Ch(node) is any node in Ch(node); h f represents the hidden state of f; S123: If the type of the node node does not belong to the statement node type set Type, then replace node with h; otherwise, vectorize the subtree T with node as the root to form a subtree vector η, and then use η to replace the subtree T in the AST; after performing the above replacement operation on all nodes in the AST, a fully structured syntax tree FS-AST is obtained; where η is calculated as follows: Perform the maximum pooling operation on the hidden state h of all nodes in T to form the subtree vector of T The calculation formula for η is: η=maxpooling(∪ g∈CH(T) h g ) Among them, CH(T) represents the set of all nodes on T; g∈CH(T) represents any node in T; h g represents the hidden state of g; ∪ g∈CH(T) h g Represents a set consisting of the hidden states of all nodes in CH(T); the maxpooling operation represents selecting the maximum value along each dimension of all hidden states in the set to form a vector.

4. A code summary generation method according to claim 1, characterized in that: The specific steps of S2 are as follows: S21: Traverse FS-AST, for each current node c that is visited, embed the hidden state h or subtree vector η of c as the node feature to form a first node feature embedding matrix Where each node h or subtree vector η is a row of X0; S22: Based on the connection relationship between each node in FS-AST, an adjacency matrix is ​​formed For any c i ,c j ∈FS-AST, the connection relationship between each node is stored in e i,j ∈E:

5. A code summary generation method according to claim 1, characterized in that: The specific steps of S3 are as follows: S31: In the first fine-tuning stage, a prompt proxy module is constructed, X0 and E are input into the prompt proxy module to obtain a prompt embedding matrix Then concatenate X0 and X1 to obtain the fused embedding matrix X2; S32: In the second fine-tuning stage, the parameters of the original CodePTMs model are expanded using the prefix vector ΔW to obtain a fine-tuned prefix vector code summary model CodePTMs' model; S33: Input X2 into the CodePTMs' model and pass it through L layers of Transformer Layers to extract semantic features and obtain the summary representation H L ; Then H L Input to the prediction head to generate a code summary; where H L Calculated by the following iterative equation: H 0 =X2 H l =TransformerLayer(H l-1 ),1≤l≤L Among them, H l Represents the output of the lth layer Transformer Layer, H l-1 Represents the output of the l-1th layer Transformer Layer, and TransformerLayer represents the calculation performed in the Transformer Layer.

6. A code summary generation method according to claim 5, characterized in that: The specific steps of S31 are as follows: S311: Construct a prompt agent module, input the first node feature embedding matrix X0 and the adjacency matrix E into the prompt agent module to obtain the prompt embedding matrix S312: Concatenate X0 and X1 to obtain the fused embedding matrix X2: X2=Concat(X1,X0) Among them, X2 contains the text information of each node on FS-AST and the structural information between nodes; The specific steps of S311 are as follows: S3111: Input X0 and E into the graph attention encoder to obtain the second node feature embedding matrix S3112: Perform flatten operation on X' to obtain the first feature representation vector β1=flatten(X') in, represents a (n×m)×1-dimensional vector space; S3113: Input β1 into the first multi-layer perceptron MLP1 for calculation to obtain the third feature representation vector S3114: Reshape β3 to obtain the hint embedding matrix X1=reshape(β3,p,m) in, Representing a p×m-dimensional vector space, the calculated X1 also contains the FS-AST structural semantics, that is, the structural information between nodes on the FS-AST; The specific steps of S3111 are as follows: S31111: Map X0 to the query space, key space, and value space respectively to obtain the query vector matrix Q, key vector matrix K, and value vector matrix V. The calculation formulas are as follows: Q=X0W Q1 ,K=X0W K1 ,V=X0W V1 Among them, the query matrix Bond Matrix Value Matrix Both are the learnable attention mechanism projection matrices in GATCoder; S31112: Calculate the attention score matrix between all nodes that make up X0 S31113: Use -∞ to mask the attention scores between non-adjacent nodes in S to obtain the masked attention score matrix S'. For any s' a,r ∈S', we have: Where node a,r∈D(X0), D(X0) represents the set of all node features that constitute X0 embedded in the corresponding nodes; s a,r ∈S represents the corresponding scores of a and r in S; s' a,r ∈S' represents the corresponding scores of a and r in S'; edge e a,r ∈E,e a,r =1 means a and r are adjacent, e a,r =0 means a and r are not adjacent, -∞ means a very small value; S31114: Perform softmax normalization on S' to obtain the attention coefficient matrix α is calculated as follows: α=softmax(S') S31115: Combine the value vector matrix V and calculate the second node feature embedding matrix X': X'=αV Among them, the calculated X' contains the FS-AST structural semantics, that is, the structural information between nodes on the FS-AST; The specific steps of S3113 are as follows: S31131: Input β1 into the first layer of the first multi-layer perceptron MLP1, perform linear transformation and activate it using the ReLU activation function to obtain the second feature representation vector β2=ReLU(W4β1+b4) S31132: Input β2 into the second layer of the first multi-layer perceptron MLP1, perform linear transformation and activate it using the same ReLU activation function as the first layer to obtain the third feature representation vector β3=ReLU(W5β2+b5) in, All are learnable parameter matrices; All are bias terms; represents a (n×m)×(n×m)-dimensional vector space, represents a (p×m)×(n×m)-dimensional vector space, represents a (n×m)×1-dimensional vector space, Represents a (p×m)×1-dimensional vector space.

7. A code summary generation method according to claim 5, characterized in that: The specific steps of S32 are as follows: S321: Calculate ΔW using low-rank matrices A and B: ΔW=AB The low-rank matrix A is an m×o-dimensional learnable parameter matrix; The low-rank matrix B is a learnable parameter matrix of dimension o×Δu; Compared with directly updating ΔW, calculating ΔW by updating the low-rank matrices A and B can improve the generalization ability and training stability of the CodePTMs' model; S322: Compare ΔW with the query matrix W in each layer of Transformer Layer Q2 , bond matrix W K2 Splicing to expand W Q2 and W K2 The dimension of is , and a fine-tuned prefix vector code summary model CodePTMs' model is obtained; the calculation formula is as follows: W’ Q2 =Concat(ΔW,W Q2 ,axis=1) W’ K2 =Concat(ΔW,W K2 ,axis=1) Concat is a concatenation operation, and the parameter axis=1 specifies that the concatenation operation is performed along the horizontal direction of the matrix.

8. A code summary generation method according to claim 1, characterized in that: The specific steps of S4 are as follows: S41: Divide into training data set, test data set and validation data set; S42: Randomly initialize the learnable parameter matrix set TD; and randomly initialize the low-rank matrices A and B in the CodePTMs' model; where TD={W Q1 ,W K1 ,W V1 ,W4,W5,b4,b5} is the set of learnable parameter matrices in the prompt agent module; S43: During the first fine-tuning phase, all learnable parameter matrices in TD are updated by performing edge prediction on FS-AST. The edge prediction task on FS-AST involves using the obtained hint embedding matrix X1 to determine whether there is a direct connection between any two nodes on FS-AST. S44: In the second fine-tuning phase, the hyperparameters set in the first fine-tuning phase are used. A suitable loss function and optimization algorithm are selected to update the parameters of the low-rank matrices A and B. The performance of the prefix vector ΔW is continuously evaluated during training, and the hyperparameters are dynamically adjusted during training to obtain a trained CodePTMs' model.

9. A code summary generation method according to claim 8, characterized in that: The specific steps of S43 are as follows: S431: For any node q, w∈D(X0), and Concatenate with the prompt embedding matrix X1 and then perform flatten operation to obtain the first structure representation vector Among them, D(X0) represents the set of all node features that constitute X0 embedded in the corresponding nodes, represents the node feature embedding corresponding to node q in X0, represents the node feature embedding corresponding to node w in X0, represents a vector space of ((p+2)×m)×1 dimensions; S432: Input z1 into the second multi-layer perceptron MLP2 to obtain the classification result S433: Set the hyperparameters required for training, including batch size, training rounds, gradient clipping threshold, learning rate, and weight decay coefficient. Then, select an appropriate loss function and optimization algorithm to update the parameters of all learnable parameter matrices in TD. During training, continuously evaluate the performance of the prompt agent module and the second multi-layer perceptron MLP2 to facilitate dynamic adjustment of hyperparameters and improve training efficiency. Finally, a trained prompt agent module is obtained. The specific steps of S432 are as follows: S4321: Input z1 into the first layer of the second multi-layer perceptron MLP2, perform linear transformation and activate it using the ReLU activation function to obtain the second structure representation vector z2=ReLU(W6z1+b6) S4322: Input z2 into the second layer of the second multi-layer perceptron MLP2, perform linear transformation and apply softmax to normalize the data to obtain the classification result. in, are all learnable parameter matrices, Both are bias terms. The structure of the second multi-layer perceptron MLP2 is exactly the same as that of the first multi-layer perceptron MLP1. The function of the second multi-layer perceptron MLP2 is to evaluate the ability of X1 to contain the structural information between the nodes on FS-AST. represents a 2×1 dimensional vector space, represents an m×1-dimensional vector space, represents an m×((p+2)×m)-dimensional vector space, Represents a 2×m-dimensional vector space.

10. A code summary generating device, characterized in that: include: The preprocessing module is used to preprocess the function source code and its corresponding code summary to obtain a multimodal function code summary dataset Conversion module for any triple Transform the FS-AST in the triple to obtain the first node feature embedding matrix and the adjacency matrix A two-stage fine-tuning design module is used to design a two-stage fine-tuning method for code summary generation tasks. It takes X0 and E as input, fine-tunes the original CodePTMs model, obtains the fine-tuned prefix vector code summary model CodePTMs' model, and outputs the code summary. Partitioning module: for multimodal function code summary dataset Divide and use it to train the CodePTMs' model; The application module is used to apply the trained CodePTMs' model to the code summary generation task and generate the corresponding code summary for the given function source code.

Images