Multi-view contrastive learning cross-language source code representation method

Abstract

The present application relates to a multi-view contrast learning cross-language source code representation method, and belongs to the technical field of computer software information. Firstly, the present application initializes the learnable parameters of the Transformer according to the programming language type using a meta-learning method; secondly, the present application constructs a code feature heterogeneous graph depending on the syntax, structure and other information of the source code segment, aggregates the heterogeneous graph using a GCN to obtain graph node embedding; then, the present application aggregates the same-hop neighborhood information of the nodes according to the meta-path to obtain the serialized representation of the graph nodes, generates the graph node embedding using a Transformer and a hierarchical attention mechanism; finally, the present application establishes a contrast loss function according to the node embedding under two perspectives, and trains a cross-language source code representation model. The present application extracts programming language specific information and constructs a model using multi-view graph node embedding to improve the cross-language code representation effect and improve the accuracy of code summarization, source code vulnerability detection and the like, in view of the fact that the existing method applies general code features of programming languages to affect the representation effect and the model appears to be over-smoothed.

Description

Technical Field

[0001] This invention relates to a multi-view comparative learning method for cross-language source code representation, belonging to the field of computer software information technology. Background Technology

[0002] Source code representation learning transforms source code into a form that machine learning models can understand and process, such as feature vectors like word embeddings and graph embeddings. It is widely used in numerous software engineering tasks, including code summarization and code completion. With the increasing scale and complexity of software and the gradual promotion of multilingual integration platforms, cross-language code conversion and reuse can greatly improve developer efficiency. Therefore, effectively constructing a unified cross-language code representation method has significant theoretical and practical value. Currently, mainstream cross-language source code representation methods can be divided into tree-based representation methods and graph-based representation methods.

[0003] 1. Tree-based representation methods

[0004] Tree-based representation methods represent source code as an Abstract Syntax Tree (AST), and then use tree-based neural network models (such as ASTNN, Tree-LSTM, etc.) to learn vector representations by recursively computing node embeddings from bottom to top. Each node in the AST represents a syntactic unit in the source code, such as a statement, expression, variable, or function declaration, while edges represent the relationships between these syntactic units, such as parent-child or sibling relationships. In cross-language code representation tasks, existing methods construct code representations based on ASTs and unify keyword representations in ASTs of different programming languages ​​through vocabulary mapping and other means. Existing cross-language code representation methods mainly learn language-independent information such as the control structure and processing logic of the code, focusing on building a unified cross-language code representation method that learns shared parameters across different languages. They pay less attention to the impact of language-specific information on the model's learning of code representations, making it difficult for the model to learn the syntactic rules and character representations of specific programming languages, thus limiting the feature extraction performance of deep learning encoders.

[0005] 2. Graph-based representation methods

[0006] Graph-based methods first construct a composite graph structure that integrates multiple code representation methods, such as abstract syntax trees, control flow graphs (CPGs), and program dependency graphs (PDGs). Then, graph neural network models (such as GAT, GCN, and GGNN) are used to extract graph features for code representation. Graph-based code representation methods use code statements and keywords as nodes, and control and dependency relationships between codes as edges. By adding or removing nodes or edges in the graph, multiple code representations can be integrated to adapt to different downstream tasks. Graph neural networks have strong local information aggregation capabilities due to their message-passing mechanism, but they are limited by the oversmoothing problem, making it difficult to effectively extract global information and restricting the model's code representation capabilities.

[0007] In summary, existing cross-language source code representation methods mainly suffer from the following problems: (1) they focus on learning a unified model of shared parameters between different languages ​​while neglecting to model language-specific information; (2) graph neural networks struggle to capture the oversmoothing problem of extracting global structural information from multi-hop neighbors. Therefore, this invention proposes a multi-view comparative learning method for cross-language source code representation. Summary of the Invention

[0008] The purpose of this invention is to address the issues of existing cross-language source code representation methods, such as the influence of common code features of applied programming languages ​​on representation performance and the problem of oversmoothing in the model, by proposing a multi-view comparative learning method for cross-language source code representation.

[0009] The design principle of this invention is as follows: First, the learnable parameters of the Transformer are initialized based on the programming language type of the input code using a meta-learning method; second, a heterogeneous graph of code features is constructed based on the syntax, structure, and other information of the source code fragments, and the graph node embeddings are obtained by aggregating the heterogeneous graph using GCN; then, the serialized representation of the graph nodes is obtained by aggregating the same-hop neighborhood information of the nodes according to the meta-path, and the graph node embeddings are generated using Transformer and hierarchical attention mechanism; finally, a contrastive loss function is established based on the node embeddings from two perspectives, and a cross-language source code representation model is trained and generated.

[0010] The technical solution of the present invention is achieved through the following steps:

[0011] Step 1: Use the meta-learning method to initialize the learnable parameters of the Transformer self-attention module according to the programming language type.

[0012] Step 2: Construct a heterogeneous graph of code features, including code syntax, control, and dependencies, from the source code fragments. Use GCN to aggregate the heterogeneous graph to obtain graph node embeddings.

[0013] Step 2.1: Use the open-source tools Joern, Soot, and ANTLR to generate AST, PDG, and CFG representations of source code in different languages.

[0014] Step 2.2: Extend the AST by incorporating PDG and CFG. Traverse the edge sets of the PDG and CFG graphs, and add edges between corresponding nodes in the AST according to the node index information at both ends, generating a heterogeneous graph with both control and dependency structural information.

[0015] Step 2.3: Perform node feature transformation to unify the feature dimensions of different types of nodes in the heterogeneous graph.

[0016] Step 2.4: Use GCN to aggregate the heterogeneous graph to obtain the node embedding from the graph perspective.

[0017] Step 3: Aggregate the same-hop neighborhood information of nodes based on the meta-path to obtain the serialized representation of the graph nodes, and use Transformer and hierarchical attention mechanism to generate node embeddings from the serialized perspective.

[0018] Step 3.1: Utilize the meta-path awareness strategy to generate the representation vector of a graph node under each meta-path. First, obtain the neighboring nodes of a node with different hop counts under different meta-paths. Then, treat neighbors from the same hop as a group, aggregate information within the group, and concatenate them to obtain the representation vector of the node under the meta-path.

[0019] Step 3.2: Use Transformer to encode the node representation vector to further explore the semantic interactions between different hop neighbors of nodes under the same metapath.

[0020] Step 3.3: Use a hierarchical attention mechanism to perform attention fusion on information within and between metapaths to generate graph node embeddings.

[0021] Step 4: Using the contrastive learning method, establish a contrastive loss function based on the node embeddings from two perspectives, and train to generate a cross-language source code representation model.

[0022] Step 4.1: Establish a set of positive and negative samples based on the correlation between nodes.

[0023] Step 4.2: Based on the node embeddings from the two perspectives, establish a contrastive loss function and a model optimization objective function, optimize them using the backpropagation algorithm, and train to generate a cross-language source code representation model.

[0024] Step 4.3: After comparative learning, the node embedding from the serialization perspective is selected as the final source code representation vector.

[0025] Beneficial effects

[0026] Compared to tree-based code representation methods, this invention generates dynamic parameters based on the language type of the input code snippet, and further generates a weight matrix of learnable parameters in the self-attention module. This method integrates language-specific information into the Transformer structure, ensuring the learning of both language-independent and language-specific information in the source code, and improving the model's cross-language source code representation performance.

[0027] Compared to graph-based code representation methods, this invention combines GCN local aggregation with meta-path global modeling capabilities, and utilizes multi-view graph node embedding to construct the model. This ensures the learning of local and global information from heterogeneous graphs of source code features, avoids the impact of oversmoothing issues in the model during graph representation learning, and improves the model's cross-language source code representation performance. Attached Figure Description

[0028] Figure 1 This is a schematic diagram illustrating the principle of the multi-view comparative learning cross-language source code representation method of this invention. Detailed Implementation

[0029] To better illustrate the purpose and advantages of the present invention, the implementation methods of the present invention will be further described in detail below with reference to examples.

[0030] The experimental data comes from Python, Ruby, Javascript, and Go code in the CodeSearchNet public dataset.

[0031] Table 1. CodeSearchNet dataset

[0032]

[0033] The specific process is as follows:

[0034] Step 1: Use the meta-learning method to initialize the learnable parameters of the Transformer self-attention module according to the programming language type.

[0035] Specifically, for a given source code fragment X i and its corresponding language type t i First, it goes through the language embedding layer. Obtain vector T i ∈R dT In the middle, then through the projection layer on T i Scaling is performed on the dimensions.

[0036]

[0037] P i =Projection(T i (2)

[0038] Where Projection(·) and P i ∈R dp These represent the embedding of the projection layer and the projection language type, respectively.

[0039] In obtaining the projection language type embedding P i Then, through the parameter generator The weight matrix W of the learnable parameters in the output attention module λ ∈R d×d .

[0040]

[0041] The method decomposes the weight representation in a manner similar to singular value decomposition, enabling the projection of the original vector onto the target matrix dimension space with fewer parameters in the generator. Diagonalization is then applied to vector P. i Transform it into a diagonal matrix, and then define two projection matrices. and The diagonalized embedding matrix is ​​projected into the space of the target weight matrix.

[0042]

[0043] Where diag(·) is a nonparametric diagonal operation, M λ and M′ λ It is a generator Learnable parameters in.

[0044] Step 2: Construct a heterogeneous graph of code features, including code syntax, control, and dependencies, from the source code fragments. Use GCN to aggregate the heterogeneous graph to obtain graph node embeddings.

[0045] Step 2.1: Use the open-source tools Joern, Soot, and ANTLR to generate AST, PDG, and CFG representations of different language codes.

[0046] Step 2.2: Extend the AST by incorporating PDG and CFG. Traverse the edge sets of the PDG and CFG graphs, and add edges between corresponding nodes in the AST according to the node index information at both ends, generating a heterogeneous graph with both control and dependency structural information.

[0047] Step 2.3: Perform node feature transformation to unify the feature dimensions of different types of nodes in the heterogeneous graph.

[0048] By creating a mapping matrix h for each node type in the heterogeneous graph i Unify node feature dimensions.

[0049]

[0050] in It is the original feature of node i. For the type of node i, yes The original feature dimension of the type node, yes Mapping matrix of type nodes, σ is the corresponding bias vector, d is the dimension of the target mapping space, and σ is the activation function.

[0051] Step 2.4: Use GCN to aggregate the heterogeneous graph to obtain the node embedding from the graph perspective.

[0052]

[0053] Where L g It is the number of layers in the graph neural network. This represents the nearest neighbor of node i under relation r. It is the feature representation of node i's nearest neighbor node j under relation r. This represents the weight matrix related to relation r. c represents the weight matrix related to the features of node i itself. i,r It is a normalization constant, h i The projection feature matrix obtained in step 2.3 This represents the feature vector updated at node i after passing through l layers of a graph convolutional neural network. g After processing by the layer network, the node embedding of node i in the graph view is obtained.

[0054] Step 3: Aggregate the co-hop neighborhood information of nodes based on the meta-path to obtain the serialized representation of the graph nodes, and use Transformer and hierarchical attention mechanism to generate node embeddings from the sequence perspective.

[0055] Step 3.1: Utilize the meta-path awareness strategy to generate the representation vector of a graph node under each meta-path. First, obtain the neighboring nodes with different hop counts under different meta-paths.

[0056] For node i, define Is node i in the metapath Let d(i,j) be the shortest path between nodes i and j, and d(i,j) be the node itself. The k-hop neighbors are N and N. 0 (i) = {i}.

[0057] Then, neighbors from the same hop are treated as a group, and information is aggregated within the group and concatenated to obtain the node's representation vector under the metapath.

[0058]

[0059] in, Metapath The adjacency matrix below, Metapath The sequence representation of the k-hop neighbors, where H is the projection feature matrix obtained in step 2.3, and D is... The degree matrix is ​​given by n, where n is the node number and d is the sequence dimension. Assuming the maximum hop count is K, for node i, the metapath can be obtained. Representation of each hop neighbor node Where k∈0,1,2,…K, concatenation yields the i-ary path of node i. The serialized representation vector below

[0060] Step 3.2: Use Transformer to encode the node representation vector to further explore the semantic interactions between different hop neighbors of nodes under the same metapath.

[0061]

[0062] in It is a learnable mapping matrix. It is of dimension d m The projection representation. The input is fed into a multi-layer encoder module to mine semantic relationships between different hop neighbors. Each encoder module consists of a multi-head attention layer (MSA), a feedforward layer (FFN), and a residual normalization layer (LN).

[0063]

[0064] in It is the intermediate output of the input sequence after passing through the residual normalization layer and the multi-head attention layer. It is the output of the multi-head attention layer The node representation is obtained after the residual normalization layer and the feedforward neural network layer. l = 1, 2, ..., L, where L is the number of encoders. After encoding through L layers, a meta-path-based node representation containing richer semantic information can be obtained.

[0065] Step 3.3: Use a hierarchical attention mechanism to perform attention fusion on information within and between metapaths to generate graph node embeddings.

[0066] Considering the information within the metapath, the weights of neighbor node information with different hop counts on the embedding representation of a metapath are obtained by calculating the correlation between neighbor nodes and the node itself.

[0067]

[0068] in, It is a learnable parameter matrix. Indicates that node i is in the metapath The k-th hop is represented below, thus realizing the information aggregation representation of neighbor nodes with different hop numbers.

[0069]

[0070] Considering the information between meta-paths, since different meta-paths express different semantics, their contribution to the final representation of a node varies across different tasks or datasets. Therefore, it is necessary to calculate the weights corresponding to the representation vectors of the same node under different meta-paths.

[0071]

[0072] Where Φ is the set of metapaths. and Metapath The learnable parameter matrix, where tanh is the activation function, ultimately yields the graph node representation from the serialization perspective.

[0073]

[0074] Step 4: Using the contrastive learning method, establish a contrastive loss function based on the node embeddings from two perspectives, and train to generate a cross-language source code representation model.

[0075] Step 4.1: Establish a set of positive and negative samples based on the correlation between nodes.

[0076] In heterogeneous graphs, different meta-paths represent different semantic relevance. If multiple meta-path instances exist between two nodes, it proves that the nodes are highly correlated. Positive and negative samples can be separated by counting the number of meta-paths between nodes.

[0077]

[0078] Among them, C i (j) represents the number of metapaths between node i and node j. Set a threshold θ; if C... i If (j) ≥ θ, add the node pair (i, j) to the positive sample set P of node i. i If the sample is in the middle, then add it to the negative sample set N. i .

[0079] Step 4.2: Based on the node embeddings from the two perspectives, establish a contrastive loss function and a model optimization objective function, optimize them using the backpropagation algorithm, and train to generate a cross-language source code representation model.

[0080]

[0081] Where sim(i,j) represents the cosine similarity between vectors i and j, and τ is the temperature coefficient. The contrastive loss represents the shift from graph-viewpoint node embeddings to serialized-viewpoint node embeddings. λ represents the contrast loss from the serialization perspective to the graph perspective, L is the overall objective function, λ is the balance coefficient between the two perspectives, and V represents the set of all graph nodes.

[0082] Step 4.3: After comparative learning, the node embedding from the serialization perspective is selected as the final source code representation vector.

Claims

1. A multi-perspective comparative learning method for cross-language source code representation, characterized by... The method includes the following steps: Step 1: Use meta-learning methods to initialize the learnable parameters of the Transformer self-attention module according to the programming language type, and introduce language-specific information into the model; Step 2: Construct a heterogeneous graph of code features, including code syntax, control, and dependencies, from the source code fragments; use GCN to aggregate the heterogeneous graph to obtain node embeddings from a graph perspective. Step 3: Aggregate the same-hop neighborhood information of nodes based on the meta-path to obtain the serialized representation of the graph nodes. Use Transformer and hierarchical attention mechanism to generate graph node embeddings: First, use the meta-path awareness strategy to generate the representation vector of the graph node under each meta-path; then use Transformer to encode the node representation vector to further explore the semantic interaction between different-hop neighborhoods of nodes under the same meta-path; finally, use the hierarchical attention mechanism to perform attention fusion on the information within and between meta-paths to generate node embeddings from the serialized perspective. Step 4: Apply the contrastive learning method to establish a contrastive loss function based on the node embeddings from two perspectives, and train to generate a cross-language source code representation model.

2. The multi-view comparative learning cross-language source code representation method according to claim 1, characterized in that: In step 1, a meta-learning method is used to generate dynamic parameters based on the programming language type of the input code snippet, and then the learnable parameters in the Transformer self-attention module are initialized, thereby introducing language-specific information learning.

3. The multi-view comparative learning cross-language source code representation method according to claim 1, characterized in that: In step 2, a heterogeneous graph of code features containing code syntax, control, and dependencies is constructed for the source code fragment. This is achieved by supplementing the Abstract Syntax Tree (AST) with the control flow graph (CPG) and program dependency graph (PDG) containing control information and dependencies.

4. The multi-view comparative learning cross-language source code representation method according to claim 1, characterized in that: In step 3, a meta-path-aware strategy is used to generate the representation vector of a graph node under each meta-path. First, the neighboring nodes with different hop counts under different meta-paths are obtained. Then, neighbors from the same hop are treated as a group, and information is aggregated within the group. in, Metapath The adjacency matrix D is... The degree matrix, Metapath The sequence representation of the k-hop neighbors is given below, where H is the projected feature matrix obtained from the node feature transformation in step 2. Assuming the maximum hop count is K, for node i, the following can be obtained in the metapath. Representation of each hop neighbor node By concatenating the elements, we can obtain the metapath of node i. The serialized representation vector below 5. The multi-view comparative learning cross-language source code representation method according to claim 1, characterized in that: In step 3, a hierarchical attention mechanism is used to perform attention fusion on information within and between metapaths. Considering information within metapaths, the importance of neighbor nodes with different hop counts to the embedding representation of a metapath is explored by calculating the correlation between neighbor nodes and the node itself, thus realizing information aggregation between neighbor nodes with different hop counts. Considering information between metapaths, since different metapaths express different semantics, the representation vectors of the same node under different metapaths are aggregated to obtain the graph node representation from the serialization perspective.

6. The cross-language source code representation method for multi-view contrastive learning according to claim 1, characterized in that: In step 4, a contrastive learning method is applied to establish a contrastive loss function based on the node embeddings from two perspectives, and to train and generate a cross-language source code representation model.

Images