Code search method, system and apparatus based on focused self-attention integrated structural component

Abstract

This invention proposes a code search method, system, and apparatus based on focused self-attention ensemble structural components, effectively improving the search accuracy for incomplete code fragments. The method employs a focused self-attention neural network to learn the semantic features of natural descriptions and code structural components hierarchically at the token, sequence, and interaction levels, thereby improving search matching accuracy. Attention projection vectors in different directions are generated through similarity matching, guiding embedded representation learning between natural descriptions and code structural components, achieving more accurate code search from local to global. This invention performs exceptionally well on public datasets, achieving a perfect score of 1 on all evaluation metrics. Even with a small training data scale, its performance significantly outperforms existing state-of-the-art baseline models. Furthermore, this invention is faster than the state-of-the-art baseline model in both model training and code query efficiency, demonstrating its advantages in both timeliness and accuracy.

Description

Technical Field

[0001] This invention relates to the field of code search in software engineering, and specifically to a code search method, system, and apparatus based on a focused self-attention integrated structure component. Background Technology

[0002] Early research typically employed information retrieval (IR) techniques, such as TF-IDF and BM25, to convert natural language descriptions into queries and retrieve relevant code from codebases. Models based on these methods often treat source code as text documents, ignoring structural information and hindering semantic matching between natural language and code. To bridge this gap, researchers proposed semantic matching models based on deep learning (DL). The DeepCS model was the first to introduce deep learning into code search, using an LSTM network to embed queries and code into the same vector space for semantic matching. Subsequently, models such as CARLCS-CNN, SAN-CS, and MESNCS improved neural code search by introducing joint attention and self-attention mechanisms to capture contextual information between code snippets and natural descriptions. Furthermore, studies such as PSCS, AdaCS, NJACS, HECS, and TCS have integrated hierarchical structure information into the code to enhance search performance.

[0003] Despite advancements in code search technology in recent years, existing methods still have several shortcomings. First, existing models such as SAN-CS and MESN-CS primarily focus on enhancing the semantic representation of queries and code, as well as their semantic interactions, while neglecting the semantic dependencies between code structural components. Code differs significantly from natural language; code structure is a more crucial component. Existing models often treat code as a concatenation of text, failing to fully capture its structured information. This limitation hinders their performance in capturing the interdependencies between code snippets, which are essential for understanding code functionality. Furthermore, many code snippets in the open-source community lack complete Abstract Syntax Tree (AST) structures, preventing previous AST-based parsers from directly extracting features for code search. Although code snippets may only contain incomplete structural components, they remain a vital resource for code reuse. Existing research, such as DeepAPI, emphasizes the importance of code components in code retrieval, but existing models still haven't addressed the interdependencies between components. Moreover, most models use fully connected concatenation methods for code components, which significantly limits their potential performance. Research shows that method names, as a concise summary of code, have a significant impact on the semantic expression of code, but existing models have failed to fully utilize this information. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a code search method based on a focused self-attention integrated structural component, which to some extent solves the problem of the lack of semantic information fusion between code structural components in existing research on code search in software question-answering communities.

[0005] To achieve the above technical solution, a code search method based on a focused self-attention integrated structure component includes the following steps:

[0006] S1. Learn word embedding representations for the structural components of the code and integrate the unified embedding representations of the code structural components through a focused self-attention neural network. The specific steps are as follows:

[0007] S1.1 Utilize a focused self-attention neural network to learn the relationship representation between words in code structure components;

[0008] Code snippets composed of word lists typically contain rich semantic information and relationships. Each code snippet consists of three distinct structural components: a method name, an API sequence, and a tag sequence. Attention weights are used to consider the positional information of these structural components to capture their respective representations.

[0009] Specifically, given Java code C = [M, A, T], assuming the "method name", "API sequence", and "tag sequence" are...

[0010]

[0011] Where H represents the length of the method name, J represents the length of the API sequence and the tag sequence, and N is a natural number; this invention makes the lengths consistent by padding the API sequence and the tag sequence with zeros;

[0012] This invention uses a randomly initialized word embedding matrix for all words in the code structure components. This matrix is ​​uniformly distributed in the range of -0.1 to 0.1 and shares a common vocabulary (Voc). code Use the following formula to obtain the word embedding of each word in the "method name":

[0013]

[0014] In the formula, It is a randomly initialized embedding matrix of dimension d. This represents the word embeddings in the "method name"; the word embedding matrix in the "method name" is then...

[0015] To achieve multi-head parallelism, dimension d must satisfy the following condition:

[0016] d = n_heads × (d k or dv )

[0017] In the formula, d is set to 128; n_heads represents the number of heads in multi-head attention, d k and d v Indicates the dimensions of the key and value vectors after the projection transformation;

[0018] S1.2, through three separate weight matrices {W Q W K W v}∈R d×d Word embedding matrix M E Projected onto the "method name" query vector "Method Name" key vector and the "method name" value vector In this context, all three vectors exist within a single dimension d, as shown in the following expression:

[0019]

[0020] In the formula, This represents the transpose of the first weight matrix. This represents the transpose of the second weight matrix. This represents the transpose of the third weight matrix;

[0021] S1.3, using a focused self-attention neural network combined with query Q name Key K name Sum V name Vector extraction of semantic information from "method name";

[0022] Unlike traditional self-attention mechanisms, this invention introduces an alignment matrix. To facilitate the comprehensive integration of relevant information, the specific expression is as follows:

[0023]

[0024] In the formula, U is the alignment matrix, specifically a normally distributed matrix with a mean of 0 and a variance of 1; Softmax(·) is the standardization function, ensuring that the sum of all column values ​​equals 1; cont name The vector contains the context matrix from which code is derived from focused self-attention.

[0025] By cont name The vector is applied to a positionally fully connected feedforward network to obtain an embedded representation of the "method name". The expression is as follows:

[0026] VT name =relu(cont)name ·W1+b1)·W2+b2

[0027] In the formula, W1∈R d’×4d’ and b1∈R 4d’ Belonging to the first group of weight matrices and bias values, W2∈R4 d×d’ , and b2∈R d’ Belongs to the second group of weight matrices and bias values; in a fully connected network, cont name It is magnified to four dimensions (4d') in the first layer, and then reduced back to one dimension (d') in the second layer by the activation function relu(·);

[0028] S1.4. By repeatedly performing the above operations, the embedding representations of the "API sequence" and the "tag sequence" are obtained, respectively.

[0029] S1.5 In order to integrate the embedded representation of code structure components, VT will be used. api VT name and VT tokens Mapped to the "API sequence" query vector respectively "Method Name" key vector and "marked sequence" value vector The specific expression is as follows:

[0030]

[0031] S1.6 After obtaining the "API sequence" query vector, the "method name" key vector, and the "tag sequence" value vector, continue to construct the code context matrix. The expression is as follows:

[0032]

[0033] By applying the context matrix to the positionally fully connected feedforward network, a unified embedding representation of the code structure components is finally obtained. This process ensures that the code embedding representation captures key information from the code structure components, while also taking into account their respective mutual attention weights;

[0034] S2. Learn word embeddings for natural descriptions and learn a unified embedding representation for natural descriptions through a focused self-attention neural network. The specific steps are as follows:

[0035] S2.1 Given a natural description, represented as Using a randomly initialized matrix Embed natural descriptions into a word embedding matrix of dimension d. middle;

[0036] S2.2 Embedding natural descriptions into the matrix Mapped to three sizes R Z×d In the vector, it is specifically represented as: Q desc K desc V desc And through three dimensions for R d×d Learnable matrix W Q W K W V To learn;

[0037] S2.3. Derive the context matrix through a focused self-attention neural network and input the matrix into a positionally fully connected feedforward network to obtain a unified embedding representation of the natural description. The expression is as follows:

[0038]

[0039] V' desc =relu(cont) desc ·W1+b1)·W2+b2;

[0040] S3. Interactive representation learning is performed on the unified embedding representation of code structure components and the unified embedding representation of natural description through a focused self-attention neural network;

[0041] The interactive embedding process emphasizes the most relevant elements and connections between natural descriptions and code components, thereby improving the accuracy and effectiveness of the search process;

[0042] We reintroduce a focused self-attention neural network to collect data during the overall embedding of code structure components and the embedding of natural descriptions. The calculation process for the (Natural Description - Code Structure Component) pair is as follows:

[0043]

[0044] In the formula, U is the alignment matrix, specifically a normally distributed matrix with a mean of 0 and a variance of 1. The alignment matrix U represents the correlation between different parts of the unified embedding representation of the natural description and each element in the unified embedding representation of the code structure component. The query vector Q, key vector K, and value vector V of the natural description and code structure component are respectively derived from V'. desc and V' code Mapped from (the calculation formula is shown in S1.5); using the weights (Softmax(Q·U·K)) inferred from the focused self-attention neural network. T ) T This generates different attention convergence directions between the unified embedding representation of code structure components and the unified embedding representation of natural descriptions, and is represented as code structure component V.C and natural description V D ;

[0045] S4. Perform average pooling on the vectors learned from the interactive representation and calculate cosine similarity for matching;

[0046] Specifically, the present invention addresses... and Perform average pooling to obtain semantic feature vectors and The expression is as follows:

[0047]

[0048] In the formula, and It is along matrix V D and V C The vector in the row direction;

[0049] The cosine similarity between each code structure component and its associated natural description is ranked to measure the relevance between the code structure component and the natural description; the higher the cosine similarity, the stronger the relationship between the natural description and the code structure component; the specific expression is as follows:

[0050]

[0051] In the formula, δ is a very small positive number, with a value of 10. -6 To avoid being divided by zero;

[0052] S5. Construct the code search model LGATCS based on a focused self-attention neural network, and construct the hinge loss function;

[0053] Specifically, the code search model LGATCS includes four modules: token-level embedding module, sequence-level embedding module, interaction-level embedding module, and similarity matching module. Among them, the token-level embedding module executes S1.1, S1.2, S1.3, S1.4, and S1.5; the sequence-level embedding module executes S1.6 and S2; the interaction-level embedding module executes S3; and the similarity matching module executes S4.

[0054] Specifically, the semantic feature vector obtained through step S4 and To construct the hinge loss function for learning model parameters;

[0055] S6. Train the hyperparameters of the code search model LGATCS, and recommend codes to users based on the obtained hyperparameter file;

[0056] S7. Experimentally evaluate the performance of the code search model LGATCS.

[0057] A code search device based on a self-attention integrated structure component includes: a code structure component semantic reorganization module 201, a natural description embedding representation module 202, a natural description and code interaction embedding module 203, and an LGATCS model code recommendation module 204.

[0058] The code structure component semantic reorganization module is used to learn word embeddings for code structure components through a focused self-attention neural network and integrate a unified representation of the structure components; this module performs the following steps:

[0059] S1. Learn word embedding representations for the structural components of the code and integrate the unified representations of the structural components through a focused self-attention neural network;

[0060] The embedding representation module for natural descriptions is used to learn word embedding representations of natural descriptions and a unified representation of natural descriptions through a focused self-attention neural network; this module performs the following steps:

[0061] S2 learns word embedding representations for natural descriptions and learns a unified representation of natural descriptions through a focused self-attention neural network;

[0062] The Natural Description and Code Interaction Embedding Module is used to establish semantic relationships between natural descriptions and code structure components; this module performs the following steps:

[0063] S3 uses a focused self-attention neural network to perform interactive representation learning on the natural description-code structure component pair, thereby obtaining the interactive representation vector of the two;

[0064] The LGATCS model code recommendation module uses the LGATCS model, based on a focused self-attention neural network, to provide corresponding code recommendations for user queries; this module performs the following steps:

[0065] S4. Perform average pooling on the interactive representation vectors of the natural description and code structure components and calculate the cosine similarity between them for matching.

[0066] S5. Construct the code search model LGATCS based on the integrated structure of a focused self-attention neural network, and construct the hinge loss function;

[0067] S6. Train the hyperparameters of the LGATCS model and recommend code to users based on the obtained model hyperparameter file;

[0068] S7. Experimentally evaluate the performance of the code search model LGATCS.

[0069] A computer-readable storage medium 300 stores a computer program 311 that, when executed, implements a code search method based on a focused self-attention integrated structure component.

[0070] A computer device, characterized in that it comprises:

[0071] Memory 401 and processor 402;

[0072] The memory 401 stores a computer program 403. When the processor 402 calls the computer program 403, it causes the processor 402 to execute a code search method that implements a self-attention integrated structure component.

[0073] Beneficial effects of the present invention

[0074] 1. An innovative method for reconstructing the semantics of incomplete structures is introduced. Unlike existing methods, this invention considers the structural semantic relationships within code structure components more deeply, rather than simply connecting the three. This method significantly improves the search accuracy of incomplete code fragments and effectively solves the challenges associated with missing or ambiguous code structures.

[0075] 2. An embedding method based on different hierarchical sequences is introduced; compared with most embedding methods that only embed at a certain level, this method improves the semantic representation of code and queries by utilizing token, sequence and interaction level embedding, capturing contextual information from local to global.

[0076] 3. This invention uses a training method based on a focused self-attention neural network model, introduces an alignment matrix U for code components to establish internal structural semantic relationships, and hierarchically integrates the semantic features of natural descriptions and code structural components. Attached Figure Description

[0077] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0078] Figure 1 A flowchart illustrating the implementation of a code search method based on a focused self-attention integrated structure component provided by this invention;

[0079] Figure 2 This invention describes the neural network model architecture and the data input and output process.

[0080] Figure 3This invention highlights the differences in neural network structure between its three-level hierarchical embedding and focused self-attention neural network and other methods.

[0081] Figure 4 This is a comparison chart showing the performance of the present invention on example training data and partial datasets compared with other models on the validation set, where (a) represents SAN-CS and SAN-CS. +EMB For the performance comparison across all data, part (b) is SAN-CS. +REI Compared with LGATCS in terms of overall data performance, part (c) is SAN-CS and SAN-CS. +EMB A comparison of performance on partial data, part (d) is SAN-CS. +REI A comparison of performance with LGATCS on some data;

[0082] Figure 5 This is a structural block diagram of the code search device based on the dedicated self-attention integrated structure component of the present invention;

[0083] Figure 6 This is a structural block diagram of a computer-readable storage medium according to the present invention;

[0084] Figure 7 This is a structural diagram of a computer device according to the present invention. Detailed Implementation

[0085] The present invention will be further described in detail below with reference to specific embodiments.

[0086] Example 1

[0087] See Figures 1-2 This embodiment provides a code search method based on a focused self-attention integrated structure component, the specific steps of which are as follows:

[0088] S1. Learn word embedding representations for the structural components of the code and integrate the unified embedding representations of the code structural components through a focused self-attention neural network. The specific steps are as follows:

[0089] S1.1 Utilize a focused self-attention neural network to learn the relationship representation between words in code structure components;

[0090] Specifically, code snippets composed of word lists typically contain rich semantic information and relationships. Each code snippet consists of three different structural components (method name, API sequence, and tag sequence), and their respective representations are captured by considering the positional information of the structural components through attention weights.

[0091] Specifically, given Java code C = [M, A, T], assuming the "method name", "API sequence", and "tag sequence" are...

[0092]

[0093] Where H represents the length of the method name, J represents the length of the API sequence and the tag sequence, and N is a natural number; this invention makes the lengths consistent by padding the API sequence and the tag sequence with zeros;

[0094] This invention uses a randomly initialized word embedding matrix for all words in the code structure components. This matrix is ​​uniformly distributed in the range of -0.1 to 0.1 and shares a common vocabulary (Voc). code Use the following formula to obtain the word embedding of each word in the "method name":

[0095]

[0096] In the formula, It is a randomly initialized embedding matrix of dimension d. This represents the word embeddings in the "method name"; the word embedding matrix in the "method name" is then...

[0097] To achieve multi-head parallelism, dimension d must satisfy the following condition:

[0098] d = n_heads × (d k or d v )

[0099] In the formula, d is set to 128; n_heads represents the number of heads in multi-head attention, d k and d v Indicates the dimensions of the key and value vectors after the projection transformation;

[0100] S1.2, through three separate weight matrices {W Q W K W v}∈R d×d Word embedding matrix M E Projected onto the "method name" query vector "Method Name" key vector and the "method name" value vector In this context, all three vectors exist within a single dimension d, as shown in the following expression:

[0101]

[0102] In the formula, This represents the transpose of the first weight matrix. This represents the transpose of the second weight matrix. This represents the transpose of the third weight matrix;

[0103] S1.3, using a focused self-attention neural network combined with query Q name Key K name Sum V name Vector extraction of semantic information from "method name";

[0104] Unlike traditional self-attention mechanisms, this invention introduces an alignment matrix. To facilitate the comprehensive integration of relevant information, the specific expression is as follows:

[0105]

[0106] In the formula, U is the alignment matrix, specifically a normally distributed matrix with a mean of 0 and a variance of 1; Softmax(·) is the standardization function, ensuring that the sum of all column values ​​equals 1; cont name The vector contains the context matrix from which code is derived from focused self-attention.

[0107] By cont name The vector is applied to a positionally fully connected feedforward network to obtain an embedding representation of the "method name". The expression is as follows:

[0108] VT name =relu(cont) name ·W1+b1)·W2+b2

[0109] In the formula, W1∈R d’×4d’ and b1∈R 4d’ Belonging to the first group of weight matrices and bias values, W2∈R 4d’×d’ , and b2∈R d’ Belongs to the second group of weight matrices and bias values; in a fully connected network, cont name It is magnified to four dimensions (4d′) in the first layer, and then reduced back to one dimension (d′) in the second layer by the activation function relu(·);

[0110] S1.4. By repeatedly performing the above operations, the embedding representations of the "API sequence" and the "tag sequence" are obtained, respectively.

[0111] S1.5 In order to integrate the embedded representation of code structure components, VT will be used. api VT name and VT tokens Mapped to the "API sequence" query vector respectively "Method Name" key vector and the "marked sequence" value vector The specific expression is as follows:

[0112]

[0113] S1.6 After obtaining the "API sequence" query vector, the "method name" key vector, and the "tag sequence" value vector, continue to construct the code context matrix. The expression is as follows:

[0114]

[0115] By applying the context matrix to the positionally fully connected feedforward network, a unified embedding representation of the code structure components is finally obtained. This process ensures that the code embedding representation captures key information from the code structure components, while also taking into account their respective mutual attention weights;

[0116] S2. Learn word embeddings for natural descriptions and learn a unified embedding representation for natural descriptions through a focused self-attention neural network. The specific steps are as follows:

[0117] S2.1 Given a natural description, represented as Using a randomly initialized matrix Embed natural descriptions into a word embedding matrix of dimension d. middle;

[0118] S2.2 Embedding natural descriptions into the matrix Mapped to three sizes R Z×d In the vector, it is specifically represented as: Q desc K desc V desc And through three dimensions for R d×d Learnable matrix W Q W K W V To learn;

[0119] Learning natural description word embeddings using a focused self-attention neural network helps capture the fundamental relationships and dependencies between different words in natural descriptions;

[0120] S2.3. Derive the context matrix through a focused self-attention neural network and input the matrix into a positionally fully connected feedforward network to obtain a unified embedding representation of the natural description. The unified embedding representation of natural descriptions integrates comprehensive context-aware semantic information, further aligning with developers' query intent. The expression is as follows:

[0121]

[0122] V' desc =relu(cont) desc ·W1+b1)·W2+b2;

[0123] S3. Interactive representation learning is performed on the unified embedding representation of code structure components and the unified embedding representation of natural description through a focused self-attention neural network;

[0124] The interactive embedding process emphasizes the most relevant elements and connections between natural descriptions and code components, thereby improving the accuracy and effectiveness of the search process;

[0125] This step, which establishes semantic relationships between natural descriptions and code structure components, is crucial for improving search accuracy.

[0126] We reintroduce a focused self-attention neural network to collect data during the overall embedding of code structure components and the embedding of natural descriptions. The calculation process for the (Natural Description - Code Structure Component) pair is as follows:

[0127]

[0128] In the formula, U is the alignment matrix, specifically a normally distributed matrix with a mean of 0 and a variance of 1. The alignment matrix U represents the correlation between different parts of the unified embedding representation of the natural description and each element in the unified embedding representation of the code structure component. The query vector Q, key vector K, and value vector V of the natural description and code structure component are respectively derived from V'. desc and V' code Mapped from (the calculation formula is shown in S1.5); using the weights (Softmax(Q·U·K)) inferred from the focused self-attention neural network. T ) T This generates different attention convergence directions between the unified embedding representation of code structure components and the unified embedding representation of natural descriptions, and is represented as code structure component V. C and natural description V D ;

[0129] S4. Perform average pooling on the vectors learned from the interactive representation and calculate cosine similarity for matching;

[0130] The purpose of average pooling is to preserve the semantic relevance of the context vector while scaling the features.

[0131] Specifically, the present invention addresses... and The semantic feature vector is obtained after performing average pooling. and The expression is as follows:

[0132]

[0133] In the formula, and It is along matrix V D and V C The vector in the row direction;

[0134] The cosine similarity between each code structure component and its associated natural description is ranked to measure the relevance between the code structure component and the natural description; the higher the cosine similarity, the stronger the relationship between the natural description and the code structure component; the specific expression is as follows:

[0135]

[0136] In the formula, δ is a very small positive number, with a value of 10. -6 To avoid being divided by zero;

[0137] S5. Construct the code search model LGATCS based on a focused self-attention neural network, and construct the hinge loss function;

[0138] Specifically, the code search model LGATCS is constructed from four modules: token-level embedding module, sequence-level embedding module, interaction-level embedding module, and similarity matching module. The token-level embedding module consists of S1.1, S1.2, S1.3, S1.4, and S1.5; the sequence-level embedding module consists of S1.6 and S2; the interaction-level embedding module consists of S3; and the similarity matching module consists of S4.

[0139] Specifically, the semantic feature vector obtained through step S4 and To construct the hinge loss function for learning model parameters;

[0140] This invention uses the open-source dataset provided by DeepCS, which is constructed as a quadruple of <method name, API sequence, tag sequence, natural description>, and training and test sets are created; the open-source dataset provides example datasets and complete datasets;

[0141] To ensure the fairness of the comparative experiments, this invention strictly adheres to the original data settings and does not include a validation set;

[0142] This invention creates triples for each data point in the training dataset of all datasets. <C,D + D - As a training example, the triplet is selected from the natural description dataset that accurately describes the relevant natural description D of code snippet C. + However, randomly selecting an irrelevant natural description D from the same dataset that cannot accurately describe code snippet C... -Composed of; if code snippets and natural description queries share similar semantics, then their respective representation vectors in the vector space are very close to each other;

[0143] When trained on the training dataset, the code search model LGATCS minimizes the loss. To make similarity The loss function expression is as follows:

[0144]

[0145] In the formula, θ represents the model parameters; P represents the training dataset; ε>0, serving as a boundary constant to avoid gradient vanishing;

[0146] S6. Train the hyperparameters of the code search model LGATCS, and recommend codes to users based on the obtained hyperparameter file;

[0147] Specifically, in the training of the code search model LGATCS, this invention uses the example dataset and the complete dataset provided by DeepCS, and their statistical information is shown in Table 1.

[0148] Table 1 shows the example and full datasets used for training and testing the LGATCS code search model.

[0149]

[0150] The specific steps are as follows:

[0151] S6.1 Input the word embeddings of the unified embedding representation of the obtained code structure components and the unified embedding representation of the natural description into the code search model LGATCS based on the focused self-attention neural network;

[0152] S6.2 In the model parameter initialization, the maximum lengths of natural description, method name, API sequence, and tag sequence are set to 30, 6, 50, and 50, respectively. The embedding dimension of word embedding and the output dimension of fully connected layer are 128. The dimensions of query and key vectors in the self-attention neural network are 16. The number of heads for multi-attention is 8. The margin value in the hinge loss function is set to 0.3986.

[0153] S6.3. Train the model using the training sets in the example dataset and the complete dataset respectively, with a learning rate of 0.0001 and the Adam optimization algorithm, until training stops after 15 iterations on the test set, and save the hyperparameter file of the code search model LGATCS.

[0154] S6.4 After training LGATCS, map all natural description-code pairs to a unified vector space;

[0155] For recommending the top k code snippets related to natural descriptions to users; where k represents the number of returned codes and k < L, and L represents the number of candidate codes in the code library G, and the value of k is 1, 5, 10; the value of the cosine similarity calculation cos(e D , e C ) represents the correlation expression between the natural description and the code snippet as follows:

[0156]

[0157] In the formula, e Cj represents the encoding of each code, j ∈ [1, g], g represents the size of the search code library G, and e D represents each natural description;

[0158] S7. Evaluate the performance of the code search model LGATCS;

[0159] Specifically, in order to illustrate the experimental effect of the code search model LGATCS of the present invention in code search in the software Q&A community, according to the hyperparameter file of the code search model LGATCS, the model performance is verified on the test set of the complete data set to give the corresponding Recall@k, MRR, NDCG code search performance evaluation indicators, and compare them with the existing baseline models DeepCS, CARLCS-CNN, CARLCS-LSTM, CSDA, SAN-CS, MESN-CS, MSARN-CS. The experimental results are shown in Table 2;

[0160] Table 2 Comparison of performance indicators of different code search baseline models

[0161]

[0162] The results show that the experimental effect of the present invention is significantly better than the other 7 baseline models. Because it re-integrates the semantic information of the code structure components based on the dedicated self-attention neural network, it is superior to other methods and can better show that the code search model LGATCS is more effective for the structure segmentation code search task; therefore, when the user gives a query, the present invention can search for more accurate codes in the software Q&A community.

[0163] In addition, in order to verify the effectiveness of the code structure component re-integration method, the present invention conducts an ablation experiment. The analysis of various baseline models shows that SAN-CS and LGATCS adopt different code concatenation methods and at the same time consistently use the Transformer architecture to achieve effective experimental comparison;

[0164] It should be noted that this invention did not select other baseline models for ablation comparison because DeepCS cannot be easily modified to adapt to self-attention networks, and CSDA, MESN-CS, and MSARN-CS are closed-source, which introduces uncertainty in their adaptability to code concatenation.

[0165] This invention employs two different code structure component connection representation methods for SAN-CS and LGATCS. ​​The fully connected method (Concatenation) is calculated using the following formula:

[0166] Tanh(W([V nam :V api :V tokens ]))

[0167] The ensemble method for the self-attention mechanism of code structure components (referred to as Attention) is calculated using the following formula:

[0168]

[0169] Experiments were conducted on the example data and partial data, and the results are shown in Table 3.

[0170] Table 3 Performance metrics for assembling code structure components using different methods

[0171]

[0172] Experimental results show that even on smaller datasets, the re-integration of code structure components has a significant impact on the model, improving the evaluation metrics by 90%.

[0173] Finally, this invention also compares the impact of Attentional Self-Attention Neural Network (AttTF) and traditional Self-Attention Neural Network (TraTF) on code search performance; firstly, AttTF is used to modify the basic embeddings and connection methods between code structure components in SAN-CS to form a comparison model SAN-CS. +EMB and SAN-CS +REI The structural differences between SAN-CS and LGATCS are shown in the attached figure. Figure 3 As shown; the present invention conducted experiments on example data, partial data and all data, and the experimental results are shown in Table 4;

[0174] Table 4 Performance metrics for different self-attention network structures

[0175]

[0176]

[0177] LGATCS achieves significant performance improvements, surpassing SAN-CS. +REI Significant enhancements to these metrics were observed in over 100% of the sample data across 15 training iterations and all data (partial training iterations) across 10 training iterations, with further performance improvements noted across the entire dataset.

[0178] In addition, such as Figure 4 As shown, SAN-CS, LGATCS, and SAN-CS are demonstrated. +EMB and SAN-CS +REI Trends in ACC and MAP code search performance evaluation metrics during model training;

[0179] The results show that the introduction of a focused self-attention neural network structure exhibits higher metric change curves in both SAN-CS and the code search model LGATCS. ​​In contrast, on a small training dataset (10,000 data points), SAN-CS combined with AttTF exhibits an atypical change trajectory. This characteristic can be attributed to the limited information learned by the model in each iteration, leading to complete underfitting of the model parameters. Essentially, the AttTF structure is indispensable for neural network model structures of various code retrieval technologies, such as SAN-CS and LGATCS.

[0180] Example 2

[0181] See details Figure 5 Based on the same inventive concept, a code search device based on a focused self-attention integrated structure component includes:

[0182] (1) Semantic reorganization module 201 for code structure components (method name, API sequence, token sequence) is used to learn word embeddings through a focused self-attention neural network and integrate the unified representation of the structure components.

[0183] This module performs the following steps: S1.

[0184] (2) Embedding representation module 202 for natural description, used to learn word embedding representations of natural description and learn a unified representation of natural description through a focused self-attention neural network.

[0185] This module performs the following steps: S2.

[0186] (3) Natural description and code interaction embedding module 203, used to establish semantic relationship between natural description and code.

[0187] This module performs the following steps: S3.

[0188] (4) Code search model LGATCS code recommendation module 204, which uses the code search model LGATCS based on a focused self-attention neural network to provide corresponding code recommendations for the user's query.

[0189] This module performs the following steps: S4, S5, S6.

[0190] Since the apparatus described in Embodiment 2 of this invention is for implementing the code search method and apparatus based on the focused self-attention integrated structure component of this invention, those skilled in the art can understand the specific structure and variations of the apparatus based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All apparatuses used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0191] The system described in Embodiment 2 of this invention is the same as the code search method based on the focused self-attention integrated structure component in Embodiment 1 of this invention. Therefore, those skilled in the art can understand the specific structure and variations of this system based on the method described in Embodiment 1 of this invention, and thus will not be repeated here. All systems employed by the method in Embodiment 1 of this invention fall within the scope of protection of this invention.

[0192] Example 3

[0193] Please see Figure 6 Based on the same inventive concept, this application also provides a computer-readable storage medium 300, on which a computer program 311 is stored, which, when executed, implements the code search method based on the focused self-attention integrated structure component as described in Embodiment 1.

[0194] Since the computer-readable storage medium described in Embodiment 3 of this invention is the same computer-readable storage medium used in implementing the code search method based on the focused self-attention integrated structure component in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer-readable storage medium based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer-readable storage media used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0195] Example 4

[0196] Based on the same inventive concept, this application also provides a computer device, please refer to [link to relevant documentation]. Figure 7 This includes: storage 401 and processor 402;

[0197] The memory 401 stores a computer program 403. When the processor calls the computer program 403, it causes the processor 402 to execute the code search method based on the focused self-attention integrated structure component as described in Embodiment 1.

[0198] Since the computer device described in Embodiment 4 of this invention is the same computer device used to implement the code search method based on the focused self-attention integrated structure component in Embodiment 1 of this invention, those skilled in the art can understand the specific structure and variations of this computer device based on the method described in Embodiment 1 of this invention, and therefore will not be repeated here. All computer devices used in the method of Embodiment 1 of this invention fall within the scope of protection of this invention.

[0199] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0201] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.

[0202] Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations to the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention also intends to include these modifications and variations.

Claims

1. A code search method based on a focused self-attention ensemble structure component, characterized in that, Includes the following steps: S1. Learn word embedding representations for the structural components of the code and integrate the unified embedding representations of the structural components of the code through a focused self-attention neural network; The specific steps are as follows: S1.1 Utilize a focused self-attention neural network to learn the relationship representation between words in the code structure components; the code structure components include: "method name", "API sequence" and "tag sequence"; S1.2, through three separate weight matrices {W Q W K W v }∈R d×d Embedding matrix M E Projected onto the "method name" query vector "Method Name" key vector and the "method name" value vector In the middle, d k Let d represent the dimension of the key after projection transformation, and H represent the length of the method name; the expression is as follows: In the formula, This represents the transpose of the first weight matrix. This represents the transpose of the second weight matrix. This represents the transpose of the third weight matrix; S1.3, using a focused self-attention neural network combined with query Q name Key K name Sum V name The semantic information of the "method name" is extracted from the vector, and the combined expression is as follows: In the formula, U is the alignment matrix, specifically a normally distributed matrix with a mean of 0 and a variance of 1; Softmax(·) is the standardization function, ensuring that the sum of all column values ​​equals 1; cont name The vector contains the context matrix from which code is derived from focused self-attention. d v Indicates the dimension of the value vector after the projection transformation; By cont name The vector is applied to a positionally fully connected feedforward network to obtain an embedding representation of the "method name". The expression is as follows: VT name =relu(cont name ·W1+b1)·W2+b2 In the formula, W1∈R d‘×4d’ and b1∈R 4d’ Belonging to the first group of weight matrices and bias values, W2∈R 4d‘×d’ , and b2∈R d’ Belongs to the second group of weight matrices and bias values; in a fully connected network, cont name It is magnified to four dimensions 4d′ in the first layer, and then reduced back to one dimension d′ in the second layer by the activation function relu(·); S1.

4. By repeatedly performing the above operations, the embedding representations of the "API sequence" and the "tag sequence" are obtained, respectively. J represents the length of the API sequence and the tag sequence; S1.5, VT api VT name and VT tokens Mapped to "API sequence" query vectors respectively "Method Name" key vector and "marked sequence" value vector The specific expression is as follows: S1.6 After obtaining the "API sequence" query vector, the "method name" key vector, and the "tag sequence" value vector, continue to construct the code context matrix. The expression is as follows: By applying the context matrix to the positionally fully connected feedforward network, a unified embedding representation of the code structure components is finally obtained. S2. Learn word embedding representations for natural descriptions and learn a unified embedding representation for natural descriptions through a focused self-attention neural network; S3. Interactive representation learning is performed on the unified embedding representation of code structure components and the unified embedding representation of natural description through a focused self-attention neural network; S4. Perform average pooling on the vectors obtained from interactive representation learning and calculate cosine similarity for matching. S5. Construct the code search model LGATCS based on a focused self-attention neural network, and construct the hinge loss function; S6. Train the hyperparameters of the code search model LGATCS, and recommend codes to users based on the obtained hyperparameter file; S7. Experimentally evaluate the performance of the code search model LGATCS.

2. The code search method based on a focused self-attention integrated structure component according to claim 1, characterized in that, In a focused self-attention neural network, an alignment matrix U is introduced for information fusion to focus on different positions. The expression for information fusion is as follows: Head (·) =Softmax(Q·U·K T ) T ·V In the formula, Q, K, and V represent the query, key, and value in self-attention, respectively.

3. The code search method based on a focused self-attention integrated structure component according to claim 1, characterized in that, The code search model LGATCS includes a token-level embedding module, a sequence-level embedding module, an interaction-level embedding module, and a similarity matching module; wherein, the token-level embedding module executes S1.1, S1.2, S1.3, S1.4 and S1.5, the sequence-level embedding module executes S1.6 and S2, the interaction-level embedding module executes S3, and the similarity matching module executes S4.

4. The code search method based on a focused self-attention integrated structure component according to claim 1, characterized in that, The specific steps for learning word embeddings for natural descriptions and learning a unified embedding representation for natural descriptions through a focused self-attention neural network are as follows: S2.

1. Given a natural description, denoted as D = d1, ..., d Z , 1≤Z≤N, Using a randomly initialized matrix Embed natural descriptions into a word embedding matrix of dimension d. middle; S2.2 Embedding natural descriptions into the matrix Mapped to three sizes R Z×d In the vector, it is specifically represented as: Q desc K desc V desc And through three dimensions for R d×d Learnable matrix W Q W K W V To learn; S2.

3. Derive the context matrix through a focused self-attention neural network and input the matrix into a positionally fully connected feedforward network to obtain a unified embedding representation of the natural description. The expression is as follows:

5. A code search device based on a focused self-attention integrated structure component, characterized in that, The code search device based on the focused self-attention integrated structure component is used to implement the code search method based on the focused self-attention integrated structure component as described in claim 1; The code search device includes: a code structure component semantic reorganization module, a natural description embedding representation module, a natural description and code interaction embedding module, and an LGATCS model code recommendation module; The code structure component semantic reorganization module is used to learn word embeddings for code structure components through a focused self-attention neural network and integrate a unified representation of the structure components; this module performs the following steps: S1. Learn word embedding representations for the structural components of the code and integrate the unified representations of the structural components through a focused self-attention neural network; The embedding representation module for natural descriptions is used to learn word embedding representations of natural descriptions and a unified representation of natural descriptions through a focused self-attention neural network; this module performs the following steps: S2 learns word embedding representations for natural descriptions and learns a unified representation of natural descriptions through a focused self-attention neural network; The Natural Description and Code Interaction Embedding Module is used to establish semantic relationships between natural descriptions and code structure components; this module performs the following steps: S3 uses a focused self-attention neural network to perform interactive representation learning on the natural description-code structure component pair, thereby obtaining the interactive representation vector of the two; The LGATCS model code recommendation module uses the LGATCS model, based on a focused self-attention neural network, to provide corresponding code recommendations for user queries; this module performs the following steps: S4. Perform average pooling on the interactive representation vectors of the natural description and code structure components and calculate the cosine similarity between them for matching. S5. Construct the code search model LGATCS based on the integrated structure of a focused self-attention neural network, and construct the hinge loss function; S6. Train the hyperparameters of the LGATCS model and recommend code to users based on the obtained model hyperparameter file; S7. Experimentally evaluate the performance of the code search model LGATCS.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the code search method based on the focused self-attention integrated structure component as described in any one of claims 1 to 4.

7. A computer device, characterized in that, include: Memory and processor; The memory stores a computer program, characterized in that when the processor invokes the computer program, the processor executes the code search method based on the focused self-attention integrated structure component as described in any one of claims 1 to 4.

Images