A multi-modal code search method based on fine-grained attention alignment

Abstract

The application discloses a multimodal code search method based on fine-grained attention alignment, constructs a new mechanism to learn the rich semantics of source code and natural language queries from two modalities of text and structure, uses a multimodal feature network to construct a code search model, adopts different construction methods for different modal features, and fully expresses different features of corresponding codes and texts; meanwhile, a fine-grained alignment mechanism is adopted in the model training process, and the same modal features are aligned respectively, thereby facilitating the matching of code entities and text entities; after the construction of all features is completed, a cosine similarity function is used to calculate the similarity between a given query and code samples in all code libraries, so that the code samples are ranked according to the similarity. The application gets rid of the insufficiency of code representation and feature extraction in the existing code search field, can more completely utilize the information contained in different modal features for code representation and matching, and has higher recognition accuracy and better convergence performance.

Description

Technical Field

[0001] This invention relates to a multimodal code search method based on fine-grained attention alignment, belonging to the field of software engineering technology. Background Technology

[0002] Software engineering is the discipline that applies engineering principles, techniques, and methods to plan, design, build, test, and maintain software systems during development, maintenance, and management. Software engineering aims to improve the efficiency, quality, and reliability of software development to meet user needs. In modern society, software systems are applied across various fields. Different software systems may contain similar functionalities, meaning their implementation code is similar. With the rapid development of the software industry in recent years, the amount of source code worldwide has reached unprecedented levels. Clearly, the software industry has entered the era of big data, and code reuse is becoming increasingly important. Code search, as a key task in the field of software engineering, aims to quickly and effectively find code snippets that match the programming semantics and the intent expressed by developers in natural language queries.

[0003] Current research indicates that code search primarily utilizes information retrieval or deep learning. Existing research methods suffer from incomplete information extraction and utilization, and insufficient consideration of feature dimensions; such information gaps significantly impact code search results. The structures and syntaxes used by code and queries differ considerably, meaning they cannot be directly or completely aligned; rapid advancements in science and technology have led to an exponential increase in code volume. In summary, the key challenges to achieving efficient code search lie in: the heterogeneity of queries and code, a unified representation of the embedded space, and the explosive growth of code databases. Summary of the Invention

[0004] Objective: To address the problems and shortcomings of existing technologies, this invention provides a multimodal code search method based on fine-grained attention alignment. This method can efficiently and quickly extract and fully align multimodal information from a large code database, resulting in better search performance. The proposed fine-grained attention alignment-based multimodal code search method can design different feature representation methods for different modal characteristics of the code itself, making the representation more accurate and thus improving the matching effect between queries and codes.

[0005] Technical Solution: A multimodal code search method based on fine-grained attention alignment. This invention proposes a fine-grained network that uses different feature representation methods for the text and structural features of code and queries. Considering feature differences, a fine-grained alignment mechanism is used to fuse features from different modalities. The method includes the following steps:

[0006] Step 1: Select code snippets from a codebase consisting of code and comments. Preprocess the code and queries to extract the corresponding text and structural representations.

[0007] Step 2: For the text features of code and query, a multi-head attention mechanism is used for representation; for the structural features of code, some data flow is added on the basis of abstract syntax tree to form a more detailed graph structure; for the structural features of query, a selection parse tree is used for representation, and NextToken edges are added to indicate its word order.

[0008] Step 3: Align the text and structural modal feature representations of the code and query respectively, and calculate the fine-grained correlations between different modal features to enhance the characteristics of the code and query.

[0009] Step 4: Connect the different modal features obtained to obtain the final representations of the code and query, optimize them using cross-entropy, and then match the code and query in Step 5.

[0010] Step 5: Use the cross-entropy optimized features from Step 4 as the final feature representations for the code and the query. Sort all code snippets according to their cosine similarity to the corresponding query. Based on the magnitude of the cosine similarity, this method finally generates a list of recommended codes corresponding to the query through this sorting.

[0011] In step 1, the code snippets selected from the standardized code database undergo preprocessing and simple extraction of their textual and structural information. Generally, the code text of all standardized code snippets can be divided into three parts: method name, API sequence, and other markers. As can be seen from the naming conventions for method names, they play a crucial role in object-oriented programming, accurately reflecting the function and behavior of the method. To reduce the complexity of method processing, a novel extraction method is innovatively used: only the method name is extracted as a feature from the code text instead of using all the code text, and tokenization is performed on the extracted tokens containing specific symbols (such as underscores). The structural information of the code is mainly extracted using tools to extract its corresponding abstract syntax tree.

[0012] In step 2, multi-head attention is used to represent the text information of the code and the query. The multi-head attention mechanism is based on the self-attention mechanism and is widely used in natural language processing. The formula for the self-attention mechanism in Transformer can be expressed as:

[0013]

[0014] in, It is a scaling factor used to correct the gradient, where Q, K, and V represent query, key, and value, respectively.

[0015] The multi-head attention mechanism repeats the self-attention mechanism process H times, and then combines the outputs:

[0016] Mutihead(Q,K,V)=Concat(head1,...,head H W Model

[0017] where head i =Attention(QW i Q ,KW i K VW i V )

[0018] in, It is the weight matrix of the attention mechanism. These are the computational parameters of the model, d Model It is W Model The corresponding matrix dimension, H, represents the number of heads. Through multi-head attention mechanisms, relationships between words in text based on the same attention mechanism can be learned, and then these relationships can be combined as knowledge to capture short-term and long-term dependencies within different ranges within a sequence.

[0019] A bidirectional gated graph neural network is used to construct the structure graph representation of code and queries. The structure graph is a heterogeneous graph, composed of different types of entities and relationships, represented as nodes and edges, respectively. Compared to traditional homogeneous graphs, heterogeneous graphs provide additional type information for nodes and edges, a feature that helps represent the different nodes and relationships contained in abstract syntax trees and selection parse trees. For each node in the structure graph, a SUM aggregation function is applied, which takes a set of incoming (or outgoing) neighboring node vectors as input and outputs a backward (or forward) aggregation vector, calculated as follows:

[0020]

[0021]

[0022] in, This represents the feature vector of the u-th neighboring node. and This represents all adjacent points in the forward and backward directions.

[0023] Embedding from forward and backward fusion nodes:

[0024]

[0025] This Fuse function calculates the gated sum of two inputs and finally provides the result to the Gated Recurrent Unit (GRU) to update the node representation:

[0026]

[0027] After K-hop calculation, the final node representation is obtained. Then max pooling is applied to all nodes to obtain the final graph representation:

[0028]

[0029] MaxPool represents the max pooling layer, and FC represents the fully connected layer.

[0030] In step 3, fine-grained alignment is performed using the two modal features of the code and query obtained in step 2. The main step is to align the structural features and textual features separately, thereby completely resolving the errors caused by feature heterogeneity in matching. A joint attention mechanism is used to weight the input sequence, enabling the model to focus on different levels of information. The key idea of ​​this mechanism is to explore finer-grained semantic relevance between the same modal features of each code and query, in order to enrich their representations respectively.

[0031] Regarding text features, the calculation process is as follows:

[0032] Attention = Code token T Query token

[0033] in and Let m and n represent the text embeddings of the code and query, respectively, where m and n represent the dimensions of the matrices. After the above calculations, the rows and columns of the Attention matrix represent relevant information about the code and query, respectively.

[0034] Use the SoftMax function to normalize the rows and columns separately:

[0035] α Code =SoftMax(Attention)

[0036] α Query =SoftMax(Attention) T )

[0037] α obtained after normalization Code and α Query These are the weight coefficients corresponding to the code and the query, respectively. The resulting weight coefficients are used to perform a dot product to obtain a new final text feature representation.

[0038] New_Query token =α Query Query token

[0039] New_Code token =α Code Code token

[0040] Regarding structural features, the calculation process is as follows:

[0041] Attention = Code structure T Query structure

[0042] in and Let m and n represent the structural embeddings of the code and query, respectively, where m and n represent the dimensions of the matrices. After the above calculations, the rows and columns of the Attention matrix represent relevant information about the code and query, respectively.

[0043] Use the SoftMax function to normalize the rows and columns separately:

[0044] α Code =SoftMax(Attention)

[0045] α Query =SoftMax(Attention) T )

[0046] α obtained after normalization Code and α Query These are the weight coefficients corresponding to the code and the query, respectively. The dot product of these weight coefficients is then used to obtain the new final structural feature representation.

[0047] New_Query structure =α Query Query structure

[0048] New_Code structure =α Code Code structure

[0049] In step 4, a joint feature representation Query is generated from the textual and structural features of the query. fusion This generates a joint feature representation of the code by combining its textual and structural features. fusion The joint feature representation is optimized using cross-entropy as the loss function.

[0050]

[0051] Among them, c i This represents the characteristic representation of the target code, c j This represents other interfering code features where i ≠ j, q i Let represent the target query features, with the base of the logarithm being e. The main objective of the loss function is to minimize q. i Its corresponding code c i The inner product between them, while maximizing q i Other interference code c j The inner product between them.

[0052] The main purpose of the loss function is to minimize q i Its corresponding code c i The inner product between them, while maximizing q i Other interference code c j The inner product between them.

[0053] Step 5 involves matching and ranking all code snippets based on the final joint feature representation of the code and query obtained in Step 4 after cross-entropy optimization. Cosine similarity is used as the matching metric, and its calculation formula is as follows:

[0054]

[0055] The higher the cosine similarity score, the better the code snippet matches the corresponding query, and the higher it ranks in the returned list.

[0056] Beneficial effects: Compared with the prior art, the multimodal code search method based on fine-grained attention alignment provided by this invention has the following advantages:

[0057] (1) The proposed fine-grained co-attention multimodal representation learning framework for code search uses only the method name in the text feature extraction of code for the first time and proves to be non-destructive to the search results, which helps to simplify matching in large code databases.

[0058] (2) Using a selection parse tree in the structural representation of natural language text is more similar to an abstract syntax tree corresponding to code structure, thus promoting fine-grained alignment.

[0059] (3) The fine-grained alignment mechanism adopted can more deeply compare the features of different modalities of code and query, and solve the problem of information overlap or redundancy caused by the heterogeneity of code and query.

[0060] (4) This invention solves the problem of insufficient information feature extraction in the field of code search, gets rid of the feature misalignment caused by the essential difference between structure and text, can select appropriate features for query matching in a large code database and improve search quality, and effectively solves some of the pain points in the field of code search. Attached Figure Description

[0061] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram of the preprocessing code in an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram of the preprocessing query in an embodiment of the present invention;

[0064] Figure 4 This is a schematic diagram illustrating the principle of the fine-grained alignment mechanism in an embodiment of the present invention. Detailed Implementation

[0065] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0066] like Figure 1 As shown, the multimodal code search method based on fine-grained attention alignment includes the following steps:

[0067] Step 1: Use a database of standard code containing <code, comment> pairs as the dataset for code search, where comments are considered their corresponding queries. Divide the code search dataset into training, testing, and validation sets in a predetermined ratio of 8:1:1. Each standard code snippet in the code dataset contains both code and query components. Figure 1 The leftmost part shown is a standard code snippet, the thin part is the corresponding code itself, and the bold part "Extracts video ID from URL." is the comment corresponding to this code, i.e., a query.

[0068] The code and queries are preprocessed separately. For the code, such as... Figure 2 As shown, the sequence of method names for extracting the text is ['def','get','vid','from','url','(','url',')'], and an abstract syntax tree corresponding to this code is constructed using a tool; regarding queries, such as Figure 3As shown, the sequence for extracting the text is ['Extracts', 'video', 'ID', 'from', 'URL'], and a selection parse tree corresponding to this natural language text is constructed using a tool.

[0069] Step 2: Different feature representation methods are used for different modalities of code and query. For the corresponding text, a multi-head attention mechanism is adopted, and the word embedding dimension is set to 128. The specific formula is as follows:

[0070] Mutihead(Q,K,V)=Concat(head1,...,head H W Model

[0071] where head i =Attention(QW i Q KW i K VW i V )

[0072] For the corresponding structure, NextToken, ReturnTo, and SubToken edges are added to the abstract syntax tree corresponding to the code, and NextToken edges are added to the parse tree of the selected region corresponding to the query. After enriching the structural information, a bidirectional gated graph neural network is used to encode and represent the structure. The dimension of the hidden layer in the graph neural network is set to 128. The specific formula is as follows:

[0073]

[0074]

[0075] Steps 3 and 4: For the text and structural modal feature representations of code and queries, calculate the fine-grained correlations between different modal features to enhance the characteristics of code and queries. Taking text features as an example, the structure of the fine-grained alignment mechanism is as follows: Figure 4 As shown.

[0076] First, perform a dot product between the obtained code and the text representation corresponding to the query:

[0077] Attention = Code token T Query token

[0078] in and Let m and n represent the text embeddings of the code and query, respectively, where m and n represent the dimensions of the matrices. After the above calculations, the rows and columns of the Attention matrix represent relevant information about the code and query, respectively.

[0079] Use the SoftMax function to normalize the rows and columns separately:

[0080] α Code =SoftMax(Attention)

[0081] α Query =SoftMax(Attention) T )

[0082] α obtained after normalization Code and α Query These are the weight coefficients corresponding to the code and the query, respectively. The resulting weight coefficients are used to perform a dot product to obtain a new final text feature representation.

[0083] New_Query token =α Query Query token

[0084] New_Code token =α Code Code token

[0085] Similarly, the final structural feature representation New_Query can be obtained. structure and New_Code structure The query's textual and structural features are combined to generate a joint feature representation. fusion This generates a joint feature representation of the code by combining its textual and structural features. fusion And it is optimized using cross-entropy as the loss function:

[0086]

[0087] Step 5: Match and sort all code snippets based on the final representation of the code obtained in Step 4 and the query. Cosine similarity is used as the matching metric, and its calculation formula is as follows:

[0088]

[0089] The results are sorted by cosine similarity and finally output as a CSV file containing the query, code, and cosine similarity score as the final code search result.

[0090] This invention trains on a standardized large-scale code database and uses a proposed multimodal code search method based on fine-grained attention alignment. First, it preprocesses the textual and structural features of both the code and the query, using different representation methods to fully extract the information contained within, avoiding information redundancy and omissions. Second, it uses a fine-grained alignment mechanism to align the features of the two modalities separately, preserving the usable information corresponding to each modality. Finally, it sorts the results using cosine similarity scores and returns the corresponding result list. This invention solves the problem of insufficient feature information for extraction and processing during matching in existing research methods, avoids the information bias caused by directly fusing features from different modalities, and can appropriately extract the required features for querying and code matching from a large code database with high accuracy.

[0091] The above are merely preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal code search method based on fine-grained attention alignment, characterized in that, The method includes the following steps: Step 1: Select code snippets from a codebase consisting of code and comments; preprocess the code and queries to extract the corresponding text and structural representations; Step 2: For the text features of the code and query, a multi-head attention mechanism is used for representation; For the structural features of the code, some data flow is added on the basis of the abstract syntax tree to form a graph structure; for the structural features of the query, a selection parse tree is used to represent it, and NextToken edges are added to indicate its word order. Step 3: Align the text and structural feature representations of the code and query respectively, and calculate the fine-grained correlation between different modal features; Step 4: Connect the different modal features obtained to obtain the final representations of the code and the query, and optimize using cross-entropy; Step 5: Use the cross-entropy optimized features from Step 4 as the final feature representations of the code and the query. Sort all code fragments according to the cosine similarity between all code fragments and the corresponding query. Based on the magnitude of the cosine similarity, this method finally obtains a list of recommended codes corresponding to the query by sorting. In step 3, fine-grained alignment is performed using the code obtained in step 2 and the two modal features of the query. The main steps are to align the structural features and text features separately, thereby completely resolving the errors caused by feature heterogeneity in the matching process. A joint attention mechanism is used to weight the input sequence, enabling the model to focus on different levels of information. Regarding text features, the calculation process is as follows: in and The text embeddings represent the code and the query, respectively. and This represents the dimension of the matrix; after the above calculation, The rows and columns of the matrix represent the code and query-related information, respectively; use The function normalizes the rows and columns separately, and the normalized result is... and These are the weight coefficients corresponding to the code and the query, respectively. The dot product of the obtained weight coefficients is used to obtain a new final text feature representation. Regarding structural features, the calculation process is as follows: in and The structural embeddings of the code and query are represented respectively. After the above calculations, The rows and columns of the matrix represent the code and query-related information, respectively; use The function normalizes the rows and columns separately, and the normalized result is... and These are the weight coefficients corresponding to the code and the query, respectively. The obtained weight coefficients are used to perform dot product calculations to obtain a new final structural feature representation.

2. The multimodal code search method based on fine-grained attention alignment according to claim 1, characterized in that, In step 1, for the code snippet selected from the code database, the code text of the code snippet is divided into three parts: method name, API sequence and other tokens. Only the method name is extracted as a feature, and the extracted tokens containing specific symbols are segmented. The abstract syntax tree corresponding to the structural information of the code is extracted.

3. The multimodal code search method based on fine-grained attention alignment according to claim 1, characterized in that, In step 2, multi-head attention is used to represent the text information of the code and the query; the self-attention mechanism in Transformer is expressed as follows: in, These are scaling factors used to correct gradients; Q, K, and V represent query, key, and value, respectively. Multi-head attention mechanisms are a repetition of the self-attention mechanism described above. Then combine the output: in, , , It is the weight matrix of the attention mechanism. These are the calculation parameters of the model. yes The corresponding matrix dimension It is the number of heads; through the multi-head attention mechanism, the relationships between words in the text based on the same attention mechanism are learned, and then they are combined as knowledge to capture short-term and long-term dependencies in different ranges within the sequence; A bidirectional gated graph neural network is used to construct the structure graph representation of code and queries; the structure graph is a heterogeneous graph, composed of different types of entities and relationships, represented by nodes and edges respectively; for each node in the structure graph, a... The aggregation function takes a set of incoming or outgoing neighbor node vectors as input and outputs a backward and forward aggregate vector. Its calculation formula is as follows: in, Indicates the first The feature vectors of the neighboring nodes and This represents all adjacent points in the forward and backward directions; Embedding from forward and backward fusion nodes: this The function calculates the gated sum of two inputs and finally provides the result to the gated loop unit to update the node representation: go through Skip the computation to obtain the final node representation. Then, max pooling is applied to all nodes to obtain the final graph representation: in Indicates the max pooling layer. This indicates a fully connected layer.

4. The multimodal code search method based on fine-grained attention alignment according to claim 1, characterized in that, In terms of text features, using The function normalizes the rows and columns separately: After normalization and These are the weight coefficients corresponding to the code and the query, respectively. The resulting weight coefficients are used to perform a dot product to obtain a new final text feature representation. 。 5. The multimodal code search method based on fine-grained attention alignment according to claim 1, characterized in that, In terms of structural features, using The function normalizes the rows and columns separately: After normalization and These are the weight coefficients corresponding to the code and the query, respectively. The dot product of these weight coefficients is then used to obtain the new final structural feature representation. 。 6. The multimodal code search method based on fine-grained attention alignment according to claim 1, characterized in that, In step 4, a joint feature representation is generated from the queried text features and structural features. This generates a joint feature representation from the textual and structural features of the code. The joint feature representation is optimized using cross-entropy as the loss function. in, This represents the characteristic representation of the target code. Indicates other interference code features and , This represents the target query feature, and the base of the logarithm is e; The main purpose of the loss function is to minimize Its corresponding code The inner product between them, while maximizing Other interference codes The inner product between them.

7. The multimodal code search method based on fine-grained attention alignment according to claim 1, characterized in that, Step 5 involves matching and ranking all code snippets based on the final joint feature representation of the code and query obtained in Step 4 after cross-entropy optimization; cosine similarity is used as the matching metric, and its calculation formula is as follows: The higher the cosine similarity score, the better the code snippet matches the corresponding query, and the higher it ranks in the returned list.

Images