Code search method, system and device based on concentrated self-attention integrated structure component

Abstract

According to the code search method, system and device based on the concentration self-attention integrated structure component, the search accuracy of incomplete code snippets is effectively improved. According to the method, a concentration self-attention neural network is adopted, semantic features of natural description and code structure components are learned hierarchically from a token level, a sequence level and an interaction level, and then search matching precision is improved. Attention projection vectors in different directions are generated through similarity matching, embedded representation learning between natural description and code structure components is guided, and more accurate code search is achieved from the local to the global. The method is excellent in performance on a public data set, the excellent score of full score 1 is obtained on all evaluation indexes, and the performance is still obviously superior to that of an existing optimal baseline model even under the condition that the training data scale is small; according to the method, the efficiency of model training and code query is higher than that of an optimal baseline model, and the advantages of the method in the aspects of timeliness and accuracy are reflected.

Description

Technical Field

[0001] The present invention relates to the field of code search in software engineering, and in particular to a code search method, system and device based on a focused self-attention integrated structural component. Background Art

[0002] Early studies usually used information retrieval (IR) techniques, such as TF-IDF and BM25, to convert natural language descriptions into query statements and retrieve relevant code from the code base. Models based on such methods usually treat source code as text documents, ignoring the structural information of the code, which hinders the semantic matching between natural language and code. To bridge this gap, researchers have proposed semantic matching models based on deep learning (DL). The DeepCS model first introduced deep learning into the field of code search, using LSTM networks to embed queries and codes into the same vector space for semantic matching. Subsequently, models such as CARLCS-CNN, SAN-CS, and MESNCS have achieved improvements in neural code search by introducing common attention and self-attention mechanisms to capture contextual information between code snippets and natural descriptions. In addition, some studies such as PSCS, AdaCS, NJACS, HECS, and TCS have also integrated hierarchical information in the code to enhance search performance.

[0003] Although code search technology has made some progress in recent years, existing methods still have many shortcomings. First, existing models such as SAN-CS and MESN-CS mainly focus on enhancing the semantic representation of queries and codes and the semantic interaction between them, but ignore the semantic correlation between code structure components. Code is significantly different from natural language, and code structure has a more important component. Existing models usually treat code as text concatenation and fail to fully capture the structural information of code. This limitation hinders the model's performance in capturing the interdependencies between code snippets, which is crucial for understanding the functionality of the code. In addition, many code snippets in the open source community lack a complete abstract syntax tree (AST) structure, which makes it impossible for previous AST-based parsers to directly extract features for code search. Although code snippets may only have incomplete code structure components, they are still an important resource for code reuse. Existing studies such as DeepAPI have emphasized the importance of code components in code retrieval, but existing models still do not address the problem of interdependencies between components. In addition, most models use a fully connected concatenation method for code components, which greatly limits their potential performance. Studies have shown that method names, as concise summaries of code, have a significant impact on the semantic expression of the code, but existing models fail to fully utilize this information. Summary of the invention

[0004] To solve the above problems, the present invention proposes a code search method based on focused self-attention integrated structural components, which to a certain extent solves the problem of lack of semantic information fusion between code structure components in code search in software question and answer communities in existing research.

[0005] In order to implement the above technical solution, a code search method based on a focused self-attention integrated structural component includes the following steps:

[0006] S1. Learn word embedding representations for the structural components of the code and integrate the unified embedding representations of the structural components of the code through a self-attention neural network. The specific steps are as follows:

[0007] S1.1. Using self-attention neural network to learn the relationship between words in code structure components;

[0008] Code snippets consisting of word lists usually contain rich semantic information and relationships. Each code snippet consists of three different structural components, including method name, API sequence, and token sequence. The position information of the structural components is considered through attention weights to capture their respective representations.

[0009] Specifically, given a Java code C = [M, A, T], assuming that the "method name", "API sequence" and "tag sequence" are

[0010]

[0011] Wherein, H represents the length of the method name, J represents the length of the API sequence and the tag sequence, and N is a natural number; the present invention makes the lengths consistent by padding the API sequence and the tag sequence with zeros;

[0012] The present invention uses a randomly initialized word embedding matrix for all words in the code structure component, which is uniformly distributed in the range of -0.1 to 0.1 and shares a common vocabulary Voc code , use the following formula to get the word embedding of each word in the method name:

[0013]

[0014] In the formula, is a randomly initialized embedding matrix of dimension d, represents the word embedding in "method name"; the word embedding matrix of "method name" is

[0015] To achieve multi-head parallelism, the dimension d must meet the following conditions:

[0016] d=n_heads×(d k or dv )

[0017] In the formula, d is set to 128; n_heads represents the number of heads in the multi-head attention, and d k and d v Represents the dimensions of the key and value vectors after the projection transformation;

[0018] S1.2, through three separate weight matrices {W Q , W K , W v}∈R d×d The word embedding matrix M E Projection to the "method name" query vector "Method Name" key vector and a vector of "method name" values In the above example, the three vectors are in a unified dimension d, and the specific expressions are as follows:

[0019]

[0020] In the formula, represents the transpose of the first weight matrix, represents the transpose of the second weight matrix, represents the transpose of the third weight matrix;

[0021] S1.3. Combining query Q with self-attention neural network name , key K name Sum value V name The vector extracts the semantic information of “method name”;

[0022] Different from the traditional self-attention mechanism, this paper introduces the alignment matrix In order to facilitate the comprehensive integration of relevant information, the specific expression is as follows:

[0023]

[0024] Where U is the alignment matrix, which is a normal distribution matrix with mean 0 and variance 1; Softmax(·) is a normalization function that ensures that the sum of all column values ​​is equal to 1; cont name The vector contains the context matrix of the code derived from the self-attention.

[0025] By adding cont name The vector is applied to the positional fully connected feed-forward network to obtain the embedding representation of "method name" The expression is as follows:

[0026] VT name =relu(contname ·W1+b1)·W2+b2

[0027] Where W1∈R d’×4d’ and b1∈R 4d’ Belongs to the first set of weight matrices and bias values, W2∈R4 d×d’ , and b2∈R d’ Belongs to the second set of weight matrices and bias values; in a fully connected network, cont name It is enlarged to four dimensions (4d') in the first layer, and then reduced back to one dimension (d') through the activation function relu (·) in the second layer;

[0028] S1.4. By repeatedly performing the above operations, the embedded representations of “API sequence” and “tag sequence” are obtained, which are

[0029] S1.5. In order to integrate the code structure components into the representation, VT api , VT name and VT tokens Mapped to the "API sequence" query vector respectively "Method Name" key vector and a vector of "tag sequence" values The specific expression is as follows:

[0030]

[0031] S1.6. After obtaining the "API sequence" query vector, the "method name" key vector, and the "tag sequence" value vector, continue to construct the code context matrix The expression is as follows:

[0032]

[0033] By applying the context matrix to the position-wise fully connected feed-forward network, we finally obtain a unified embedding representation of the code structure components. This process ensures that the code embedding representation captures key information from the code structural components while also considering their respective mutual attention weights;

[0034] S2. Learn word embedding representations for natural descriptions and learn unified embedding representations for natural descriptions through self-attention neural networks. The specific steps are as follows:

[0035] S2.1. Given a natural description, expressed as Using randomly initialized matrices Embed the natural description into a word embedding matrix of dimension d middle;

[0036] S2.2. Embedding Natural Description into Matrix Mapped to three sizes R Z×d In the vector, it is specifically expressed as: Q desc , K desc , V desc , and through three dimensions for R d×d The learnable matrix W Q , W K , W V To study;

[0037] S2.3. Derive the context matrix through the self-attention neural network and input the matrix into the position fully connected feedforward network to obtain a unified embedding representation of natural descriptions The expression is as follows:

[0038]

[0039] V' desc =relu(cont desc ·W1+b1)·W2+b2;

[0040] S3, interactive representation learning of unified embedding representations of code structure components and natural descriptions through attentive self-attention neural networks;

[0041] The interactive embedding process highlights the most relevant elements and connections between natural descriptions and code components, thus improving the precision and effectiveness of the search process;

[0042] The self-attention neural network is introduced again to collect (Natural Description-Code Structure Component) pair, the calculation process is as follows:

[0043]

[0044] Where U is the alignment matrix, specifically a normal distribution matrix with mean 0 and variance 1. The alignment matrix U represents the correlation between different parts of the unified embedding representation of the natural description and each element in the unified embedding representation of the code structure component. The query vector Q, key vector K and value vector V of the natural description and code structure component are represented by V' desc and V' code Mapped (the calculation formula is shown in S1.5); using the weights inferred from the self-attention neural network (Softmax (Q·U·K T ) T ), different attention pooling directions are generated between the unified embedding representation of the code structure component and the unified embedding representation of the natural description and represented as the code structure component VC and natural description V D ;

[0045] S4, performing an average pooling operation on the vectors of interactive representation learning and calculating cosine similarity for matching;

[0046] Specifically, the present invention is and Perform an average pooling operation to obtain the semantic feature vector and The expression is as follows:

[0047]

[0048] In the formula, and is along the matrix V D and V C The row direction vector of ;

[0049] The cosine similarity between each code structure component and the relevant natural description is ranked to measure the correlation between the code structure component and the natural description; the higher the cosine similarity, the greater the relationship between the natural description and the code structure component; the specific expression is as follows:

[0050]

[0051] In the formula, δ is a very small positive number, with a value of 10 -6 , to avoid division by zero;

[0052] S5. Construct a code search model LGATCS based on self-attention neural network and construct a hinge loss function;

[0053] Specifically, the code search model LGATCS includes four modules: token-level embedding module, sequence-level embedding module, interaction-level embedding module, and similarity matching module; among them, the token-level embedding module executes S1.1, S1.2, S1.3, S1.4, and S1.5, the sequence-level embedding module executes S1.6 and S2, the interaction-level embedding module executes S3, and the similarity matching module executes S4;

[0054] Specifically, the semantic feature vector obtained in step S4 and to construct the hinge loss function for learning model parameters;

[0055] S6. Train the hyperparameters of the code search model LGATCS, and recommend codes to user queries based on the obtained hyperparameter files;

[0056] S7. Experimental evaluation of the performance of the code search model LGATCS.

[0057] A code search device based on focused self-attention integrated structural components, comprising: a code structural component semantic reorganization module 201, a natural description embedding representation module 202, a natural description and code interactive embedding module 203, and a LGATCS model code recommendation module 204;

[0058] The code structure component semantic reorganization module is used to learn the word embedding of the code structure components through the self-attention neural network and integrate the unified representation of the structure components; this module performs the following steps:

[0059] S1. Learn word embedding representations for the structural components of the code and integrate the unified representation of the structural components through a self-attention neural network;

[0060] The embedding representation module of natural description is used to learn the word embedding representation of natural description and learn the unified representation of natural description through the self-attention neural network; this module performs the following steps:

[0061] S2 learns word embedding representations for natural descriptions and learns a unified representation of natural descriptions through a self-attention neural network;

[0062] The natural description and code interaction embedding module is used to establish the semantic relationship between the natural description and code structure components; this module performs the following steps:

[0063] S3 uses a self-attention neural network to interactively learn the natural description-code structure component pairs and obtain the interactive representation vectors of the two.

[0064] The LGATCS model code recommendation module uses the LGATCS model based on the self-attention neural network to give corresponding code recommendations for the user's query; this module performs the following steps:

[0065] S4, performing an average pooling operation on the interactive representation vectors of the natural description and the code structure component and calculating the cosine similarity between the two for matching;

[0066] S5. Construct a code search model LGATCS based on the integrated structural component of the self-attention neural network and construct a hinge loss function;

[0067] S6. Train the hyperparameters of the LGATCS model and recommend codes to user queries based on the obtained model hyperparameter file;

[0068] S7. Experimental evaluation of the performance of the code search model LGATCS.

[0069] A computer-readable storage medium 300 stores a computer program 311 thereon, which, when executed, implements a code search method based on a focused self-attention integrated structural component.

[0070] A computer device, comprising:

[0071] Memory 401 and processor 402;

[0072] The memory 401 stores a computer program 403. When the processor 402 calls the computer program 403, the processor 402 executes a code search method based on a focused self-attention integrated structural component.

[0073] Beneficial Effects of the Invention

[0074] 1. An innovative method for reconstructing incomplete structural semantics is introduced. Different from existing methods, the present invention takes into account the structural semantic connections within the code structure components more deeply, rather than just connecting the three. This method significantly improves the search accuracy of incomplete code snippets and effectively solves the challenges associated with missing or ambiguous code structures.

[0075] 2. An embedding method based on sequences at different levels is introduced. Compared with most embedding methods that only embed a certain level, this method improves the semantic representation of code and queries by utilizing embeddings at the token, sequence, and interaction levels, capturing contextual information from local to global.

[0076] 3. The present invention uses a training method based on a focused self-attention neural network model, introduces an alignment matrix U for code components to establish internal structural semantic connections, and hierarchically integrates natural descriptions and semantic features of code structure components. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0078] Figure 1 A flowchart of a code search method based on a focused self-attention integrated structural component provided by the present invention;

[0079] Figure 2 The neural network model architecture and data input and output process of the present invention;

[0080] Figure 3The differences in the neural network structure between the three-level embedding and self-attention neural network of the present invention and other methods;

[0081] Figure 4 This is a comparison diagram of the effect of the present invention on the example training data and part of the data set with other models on the validation set, where (a) is SAN-CS and SAN-CS +EMB Comparison of the effects on all data, part (b) is SAN-CS +REI Comparison of the effects of LGATCS on all data. Part (c) shows SAN-CS and SAN-CS. +EMB Comparison of the effects on some data, part (d) is SAN-CS +REI Comparison with LGATCS on some data;

[0082] Figure 5 It is a structural block diagram of a code search device based on a focused self-attention integrated structural component of the present invention;

[0083] Figure 6 A structural block diagram of a computer-readable storage medium of the present invention;

[0084] Figure 7 The present invention is a structural diagram of a computer device. DETAILED DESCRIPTION

[0085] The present invention is further described in detail below in conjunction with specific embodiments.

[0086] Embodiment 1

[0087] See also Figure 1-Figure 2 This embodiment provides a code search method based on a focused self-attention integrated structural component, and the specific steps are as follows:

[0088] S1. Learn word embedding representations for the structural components of the code and integrate the unified embedding representations of the structural components of the code through a self-attention neural network. The specific steps are as follows:

[0089] S1.1. Using self-attention neural network to learn the relationship between words in code structure components;

[0090] Specifically, code snippets consisting of word lists usually contain rich semantic information and relations. Each code snippet consists of three different structural components (method name, API sequence, and token sequence). The position information of the structural components is considered through attention weights to capture their respective representations.

[0091] Specifically, given a Java code C = [M, A, T], assuming that the "method name", "API sequence" and "tag sequence" are

[0092]

[0093] Wherein, H represents the length of the method name, J represents the length of the API sequence and the tag sequence, and N is a natural number; the present invention makes the lengths consistent by padding the API sequence and the tag sequence with zeros;

[0094] The present invention uses a randomly initialized word embedding matrix for all words in the code structure component, which is uniformly distributed in the range of -0.1 to 0.1 and shares a common vocabulary Voc code , use the following formula to get the word embedding of each word in the method name:

[0095]

[0096] In the formula, is a randomly initialized embedding matrix of dimension d, represents the word embedding in "method name"; the word embedding matrix of "method name" is

[0097] To achieve multi-head parallelism, the dimension d must meet the following conditions:

[0098] d=n_heads×(d k or d v )

[0099] In the formula, d is set to 128; n_heads represents the number of heads in the multi-head attention, and d k and d v Represents the dimensions of the key and value vectors after the projection transformation;

[0100] S1.2, through three separate weight matrices {W Q , W K , W v}∈R d×d The word embedding matrix M E Projection to the "method name" query vector "Method Name" key vector and a vector of "method name" values In the above example, the three vectors are in a unified dimension d, and the specific expressions are as follows:

[0101]

[0102] In the formula, represents the transpose of the first weight matrix, represents the transpose of the second weight matrix, represents the transpose of the third weight matrix;

[0103] S1.3. Combining query Q with self-attention neural network name , key K name Sum value V name The vector extracts the semantic information of “method name”;

[0104] Different from the traditional self-attention mechanism, this paper introduces the alignment matrix In order to facilitate the comprehensive integration of relevant information, the specific expression is as follows:

[0105]

[0106] Where U is the alignment matrix, which is a normal distribution matrix with mean 0 and variance 1; Softmax(·) is a normalization function that ensures that the sum of all column values ​​is equal to 1; cont name The vector contains the context matrix of the code derived from the self-attention.

[0107] By adding cont name The vector is applied to the positional fully connected feed-forward network to obtain the embedding representation of "method name" The expression is as follows:

[0108] VT name =relu(cont name ·W1+b1)·W2+b2

[0109] Where W1∈R d’×4d’ and b1∈R 4d’ Belongs to the first set of weight matrices and bias values, W2∈R 4d’×d’ , and b2∈R d’ Belongs to the second set of weight matrices and bias values; in a fully connected network, cont name It is enlarged to four dimensions (4d′) in the first layer, and then reduced back to one dimension (d′) through the activation function relu(·) in the second layer;

[0110] S1.4. By repeatedly performing the above operations, we obtain the embedded representations of “API sequence” and “tag sequence”, which are

[0111] S1.5. In order to integrate the code structure components into the representation, VT api , VT name and VT tokens Mapped to the "API sequence" query vector respectively "Method Name" key vector and a vector of "tag sequence" values The specific expression is as follows:

[0112]

[0113] S1.6. After obtaining the "API sequence" query vector, the "method name" key vector, and the "tag sequence" value vector, continue to construct the code context matrix The expression is as follows:

[0114]

[0115] By applying the context matrix to the position-wise fully connected feed-forward network, we finally obtain a unified embedding representation of the code structure components. This process ensures that the code embedding representation captures key information from the code structural components while also considering their respective mutual attention weights;

[0116] S2. Learn word embedding representations for natural descriptions and learn unified embedding representations for natural descriptions through self-attention neural networks. The specific steps are as follows:

[0117] S2.1. Given a natural description, represented as Using randomly initialized matrices Embed the natural description into a word embedding matrix of dimension d middle;

[0118] S2.2. Embedding Natural Description into Matrix Mapped to three sizes R Z×d In the vector, it is specifically expressed as: Q desc , K desc , V desc , and through three dimensions for R d×d The learnable matrix W Q , W K , W V To study;

[0119] Learning natural description word embedding representations through self-attention neural networks helps capture the basic relationships and dependencies between different words in natural descriptions;

[0120] S2.3. Derive the context matrix through the self-attention neural network and input the matrix into the position fully connected feedforward network to obtain a unified embedding representation of natural descriptions The unified embedding representation of natural descriptions integrates comprehensive context-aware semantic information, further conforming to the developer’s query intent, as expressed as follows:

[0121]

[0122] V' desc =relu(cont desc ·W1+b1)·W2+b2;

[0123] S3, interactive representation learning of unified embedding representations of code structure components and natural descriptions through attentive self-attention neural networks;

[0124] The interactive embedding process highlights the most relevant elements and connections between natural descriptions and code components, thus improving the precision and effectiveness of the search process;

[0125] This step establishes semantic relationships between natural descriptions and code structure components and is crucial for improving search accuracy;

[0126] The self-attention neural network is introduced again to collect (Natural Description-Code Structure Component) pair, the calculation process is as follows:

[0127]

[0128] Where U is the alignment matrix, specifically a normal distribution matrix with mean 0 and variance 1. The alignment matrix U represents the correlation between different parts of the unified embedding representation of the natural description and each element in the unified embedding representation of the code structure component. The query vector Q, key vector K and value vector V of the natural description and code structure component are represented by V' desc and V' code Mapped (the calculation formula is shown in S1.5); using the weights inferred from the self-attention neural network (Softmax (Q·U·K T ) T ), different attention pooling directions are generated between the unified embedding representation of the code structure component and the unified embedding representation of the natural description and represented as the code structure component V C and natural description V D ;

[0129] S4, performing an average pooling operation on the vectors of interactive representation learning and calculating cosine similarity for matching;

[0130] The role of the average pooling operation is to use feature scaling while also preserving the semantic relevance of the context vector;

[0131] Specifically, the present invention is and After performing the average pooling operation, the semantic feature vector is obtained and The expression is as follows:

[0132]

[0133] In the formula, and is along the matrix V D and V C The row direction vector of ;

[0134] The cosine similarity between each code structure component and the relevant natural description is ranked to measure the correlation between the code structure component and the natural description; the higher the cosine similarity, the greater the relationship between the natural description and the code structure component; the specific expression is as follows:

[0135]

[0136] In the formula, δ is a very small positive number, with a value of 10 -6 , to avoid division by zero;

[0137] S5. Construct a code search model LGATCS based on self-attention neural network and construct a hinge loss function;

[0138] Specifically, the code search model LGATCS is constructed by combining four modules: token-level embedding module, sequence-level embedding module, interaction-level embedding module, and similarity matching module; the token-level embedding module is composed of S1.1, S1.2, S1.3, S1.4, and S1.5, the sequence-level embedding module is composed of S1.6 and S2, the interaction-level embedding module is composed of S3, and the similarity matching module is composed of S4;

[0139] Specifically, the semantic feature vector obtained in step S4 and to construct the hinge loss function for learning model parameters;

[0140] The present invention uses an open source dataset provided by DeepCS, which is constructed as a four-tuple of <method name, API sequence, tag sequence, natural description>, and creates a training set and a test set; the open source dataset provides an example dataset and a complete dataset;

[0141] In order to ensure the fairness of the comparative experiment, this paper strictly follows the original data setting and does not include the validation set;

[0142] The present invention creates a triplet for each piece of data in the training data set of all data sets. <C,D + , D - >As a training example; the triplet is selected from the natural description data set to accurately describe the code snippet C related natural description D + , and randomly select an irrelevant natural description D from the same dataset that cannot accurately describe the code snippet C -If the code snippet and the natural description query share similar semantics, then their respective representation vectors in the vector space are very close to each other;

[0143] When trained on the training dataset, the code search model LGATCS minimizes the loss To make similarity The loss function expression is as follows:

[0144]

[0145] Where θ represents the model parameters; P represents the training data set; ε>0, which is a boundary constant to avoid gradient vanishing;

[0146] S6. Train the hyperparameters of the code search model LGATCS, and recommend codes to user queries based on the obtained hyperparameter files;

[0147] Specifically, in the training of the code search model LGATCS, the present invention uses the example data set and the complete data set provided by DeepCS, and its statistical information is shown in Table 1;

[0148] Table 1. Example datasets and complete datasets used for training and testing of the code search model LGATCS

[0149]

[0150] The specific steps are as follows:

[0151] S6.1. Input the obtained unified embedding representation of the code structure components and the word embedding of the unified embedding representation of the natural description into the code search model LGATCS based on the focused self-attention neural network;

[0152] S6.2, in the model parameter initialization, the maximum lengths of natural description, method name, API sequence, and tag sequence are set to 30, 6, 50, and 50 respectively, the embedding dimension of word embedding and the output dimension of the fully connected layer are 128, the dimension of the query and key vectors in the self-attention neural network is 16, the number of heads of multi-attention is 8, and the margin value in the hinge loss function is set to 0.3986;

[0153] S6.3. Perform training according to the training sets in the example dataset and the complete dataset respectively, with a learning rate of 0.0001 and the Adam optimization algorithm, until the training is stopped after 15 iterations of the test set, and save the hyperparameter file of the code search model LGATCS;

[0154] S6.4, after training LGATCS, map all natural description-code pairs to a unified vector space;

[0155] For recommending the top k code snippets related to the natural description to the user; where k represents the number of returned codes and k < L, and L represents the number of candidate codes in the code library G, and the value of k is 1, 5, 10; the value of the cosine similarity calculation cos(e D , e C ) represents the correlation expression between the natural description and the code snippet as follows:

[0156]

[0157] In the formula, e Cj represents the encoding of each code, j ∈ [1, g], g represents the size of the searched code library G, and e D represents each natural description;

[0158] S7. Experimentally evaluate the performance of the code search model LGATCS;

[0159] Specifically, in order to illustrate the experimental effect of the code search model LGATCS of the present invention on code search in the software Q&A community, according to the hyperparameter file of the code search model LGATCS, model performance verification is carried out on the test set of the complete data set to give the corresponding Recall@k, MRR, NDCG code search performance evaluation indicators, and compare them with the existing baseline models DeepCS, CARLCS-CNN, CARLCS-LSTM, CSDA, SAN-CS, MESN-CS, MSARN-CS. The experimental results are shown in Table 2;

[0160] Table 2 Comparison of performance indicators of different code search baseline models

[0161]

[0162] The results show that the experimental effect of the present invention is significantly better than the other 7 baseline models. Because it re-integrates the semantic information of the code structure components based on the dedicated self-attention neural network, it is superior to other methods and can better show that the code search model LGATCS is more effective for the structure segmentation code search task; therefore, when the user gives a query, the present invention can search for more accurate codes in the software Q&A community.

[0163] In addition, in order to verify the effectiveness of the code structure component re-integration method, the present invention conducts ablation experiments. The analysis of various baseline models shows that SAN-CS and LGATCS adopt different code concatenation methods and simultaneously use the Transformer architecture consistently, thus realizing an effective experimental comparison;

[0164] It should be noted that the present invention does not select other baseline models for ablation comparison because DeepCS cannot be easily modified to adapt to the self-attention network, and CSDA, MESN-CS and MSARN-CS are closed-source, which introduces uncertainty in their adaptability to code concatenation;

[0165] The present invention adopts two different code structure component connection representation methods for SAN-CS and LGATCS, wherein the full connection method (abbreviated as Concatenation) has the following calculation formula:

[0166] Tanh(W([V nam :V api :V tokens ]))

[0167] The self-attention mechanism integration method of the code structure component (abbreviated as Attention) is calculated as follows:

[0168]

[0169] Experiments are conducted on sample data and partial data, and the experimental results are shown in Table 3;

[0170] Table 3 Performance indicators of using different methods to splice code structure components

[0171]

[0172] Experimental results show that even on smaller datasets, the reintegration of code structure components has a significant impact on the model, improving the evaluation index by 90%;

[0173] Finally, the present invention also compares the effects of the attentive self-attention neural network (AttTF) and the traditional self-attention neural network (TraTF) on code search performance; firstly, AttTF is used to modify the basic embedding in SAN-CS and the connection method between code structure components to form a comparison model SAN-CS +EMB and SAN-CS +REI ; The structural differences between SAN-CS and LGATCS are shown in the attached Figure 3 As shown; the present invention has been experimented on sample data, partial data and all data, and the experimental results are shown in Table 4;

[0174] Table 4 Performance indicators using different self-attention network structures

[0175]

[0176]

[0177] LGATCS achieves significant performance improvement, surpassing SAN-CS +REI More than 100%, significant enhancements in these metrics were observed on the example data for a total of 15 training iterations and all data for 10 training iterations (partial training iterations), and further performance improvements in each metric were noted on the entire dataset;

[0178] In addition, if Figure 4 As shown, SAN-CS, LGATCS and SAN-CS are shown. +EMB and SAN-CS +REI Trends in the ACC and MAP code search performance evaluation metrics during model training;

[0179] The results show that the introduction of the self-attention neural network structure shows a higher metric change curve in both SAN-CS and the code search model LGATCS; in contrast, on a small training dataset (10,000 data points), SAN-CS combined with AttTF shows an atypical change trajectory, which can be attributed to the limited information learned by the model at each iteration, resulting in the model's parameters being completely underfit; essentially, the AttTF structure is indispensable for various neural network model structures for code retrieval, such as SAN-CS and LGATCS;

[0180] Embodiment 2

[0181] See Figure 5 , based on the same inventive concept, a code search device based on a focused self-attention integrated structural component, comprising:

[0182] (1) Code structure component semantic reorganization module 201, used to learn word embedding of code structure components (method name, API sequence, token sequence) through self-attention neural network, and integrate the unified representation of structure components.

[0183] This module performs the following steps: S1.

[0184] (2) A natural description embedding representation module 202, which is used to learn the word embedding representation of natural descriptions and learn the unified representation of natural descriptions through a self-attention neural network.

[0185] This module performs the following steps: S2.

[0186] (3) A natural description and code interactive embedding module 203, used to establish a semantic relationship between the natural description and the code.

[0187] This module performs the following steps: S3.

[0188] (4) Code search model LGATCS code recommendation module 204 uses the code search model LGATCS based on the focused self-attention neural network to give corresponding code recommendations for the user's query.

[0189] This module performs the following steps: S4, S5, S6.

[0190] Since the device introduced in the second embodiment of the present invention is a code search method and device based on the focused self-attention integrated structural component of the embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the device, so it is not repeated here. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0191] The system introduced in the second embodiment of the present invention is a system of the code search method based on the self-attention integrated structural component in the first embodiment of the present invention. Therefore, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the system, so it is not repeated here. All systems adopted by the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0192] Embodiment 3

[0193] See also Figure 6 Based on the same inventive concept, the present application also provides a computer-readable storage medium 300 on which a computer program 311 is stored. When the computer program 311 is executed, the code search method based on the focused self-attention integrated structural component as described in Example 1 is implemented.

[0194] Since the computer-readable storage medium introduced in the third embodiment of the present invention is a computer-readable storage medium used to implement the code search method based on the focused self-attention integrated structural component in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer-readable storage medium, so it is not repeated here. All computer-readable storage media used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0195] Embodiment 4

[0196] Based on the same inventive concept, the present application also provides a computer device, see Figure 7 , including: storage 401, processor 402;

[0197] The memory 401 stores a computer program 403. When the processor calls the computer program 403, the processor 402 executes the code search method based on the focused self-attention integrated structural component as described in Example 1.

[0198] Since the computer device introduced in the fourth embodiment of the present invention is a computer device used to implement the code search method based on the focused self-attention integrated structural component in the first embodiment of the present invention, based on the method introduced in the first embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the computer device, so it is not repeated here. All computer devices used in the method in the first embodiment of the present invention belong to the scope of protection of the present invention.

[0199] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0200] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0202] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A code search method based on self-attention integrated structure components, characterized in that: The following steps are involved: S1. Learn word embedding representations for the structural components of the code and integrate the unified embedding representation of the structural components of the code through a self-attention neural network; S2, learning word embedding representations for natural descriptions and learning unified embedding representations of natural descriptions through self-attention neural networks; S3, interactive representation learning of unified embedding representations of code structure components and natural descriptions through attentive self-attention neural networks; S4, performing average pooling operation on the vectors obtained by interactive representation learning and calculating cosine similarity for matching; S5. Construct a code search model LGATCS based on self-attention neural network and construct a hinge loss function; S6. Train the hyperparameters of the code search model LGATCS, and recommend codes to user queries based on the obtained hyperparameter files; S7. Experimental evaluation of the performance of the code search model LGATCS.

2. A code search method based on focused self-attention integrated structural components according to claim 1, characterized in that: The alignment matrix U is introduced into the self-attention neural network for information fusion to focus on different positions. The expression of information fusion is as follows: Head (·) =Softmax(Q·U·K T ) T ·V Where Q, K, and V represent the query, key, and value in self-attention, respectively.

3. A code search method based on focused self-attention integrated structural components according to claim 1, characterized in that: The code search model LGATCS includes a token-level embedding module, a sequence-level embedding module, an interaction-level embedding module, and a similarity matching module; wherein the token-level embedding module executes S1.1, S1.2, S1.3, S1.4, and S1.5, the sequence-level embedding module executes S1.6 and S2, the interaction-level embedding module executes S3, and the similarity matching module executes S4.

4. A code search method based on focused self-attention integrated structural components according to claim 1, characterized in that: Learn word embedding representations for the structural components of the code and integrate the unified embedding representations of the structural components of the code through a self-attention neural network. The specific steps are as follows: S1.

1. Using a self-attention neural network to learn the relationship representation between words in code structure components; the code structure components include: "method name", "API sequence" and "tag sequence"; S1.2, through three separate weight matrices {W Q , W k , W v }∈R d×d The embedding matrix M E Projection to the "method name" query vector "Method Name" key vector and a vector of "method name" values Medium, d k represents the dimension of the key after projection transformation, and d represents the dimension; the expression is as follows: In the formula, represents the transpose of the first weight matrix, represents the transpose of the second weight matrix, represents the transpose of the third weight matrix; S1.

3. Combining query Q with self-attention neural network name , key K name Sum value V name The vector extracts the semantic information of "method name", and the combined expression is as follows: Where U is the alignment matrix, which is a normal distribution matrix with mean 0 and variance 1; Softmax(·) is a normalization function that ensures that the sum of all column values ​​is equal to 1; cont name The vector contains the context matrix of the code derived from the self-attention. d v Represents the dimension of the value vector after projection transformation; By adding cont name The vector is applied to the positional fully connected feed-forward network to obtain the embedded representation of "method name" The expression is as follows: VT name =relu(cont name ·W1+b1)·W2+b2 Where W1∈R d‘×4d’ and b1∈R 4d’ Belongs to the first set of weight matrices and bias values, W2∈R 4d‘×d’ , and b2∈R d’ Belongs to the second set of weight matrices and bias values; in a fully connected network, cont name It is enlarged to four dimensions (4d') in the first layer, and then reduced back to one dimension (d') through the activation function relu (·) in the second layer; S1.

4. By repeatedly performing the above operations, the embedded representations of "API sequence" and "label sequence" are obtained, which are S1.5、VT api , VT name and VT tokens Mapped to the "API sequence" query vector respectively "Method Name" key vector and a vector of "tag sequence" values The specific expression is as follows: S1.

6. After obtaining the "API sequence" query vector, the "method name" key vector, and the "tag sequence" value vector, continue to construct the code context matrix The expression is as follows: By applying the context matrix to the position-wise fully connected feed-forward network, we finally obtain a unified embedding representation of the code structure components.

5. A code search method based on focused self-attention integrated structural components according to claim 1, characterized in that: The specific steps of learning word embedding representation for natural descriptions and learning unified embedding representation of natural descriptions through self-attention neural network are as follows: S2.

1. Given a natural description, denoted as D = d1, ..., d Z , 1≤Z≤N, Using randomly initialized matrices Embed the natural description into a word embedding matrix of dimension d middle; S2.

2. Embedding Natural Description into Matrix Mapped to three sizes R Z×d In the vector, it is specifically expressed as: Q desc , K desc , V desc , and through three dimensions for R d×d The learnable matrix W Q , W K , W V To study; S2.

3. Derive the context matrix through the self-attention neural network and input the matrix into the position fully connected feedforward network to obtain a unified embedding representation of natural descriptions The expression is as follows: <h2 style=";text-align:left;direction:ltr">V'<h2 style=";text-align:left;direction:ltr"> desc <h2 style=";text-align:left;direction:ltr"> (cont)<h2 style=";text-align:left;direction:ltr"> desc <h2 style=";text-align:left;direction:ltr"> ·W1+b1)·W2+b2。 6. A code search device based on a focused self-attention integrated structural component, characterized in that: The code search device comprises: a code structure component semantic reorganization module, a natural description embedding representation module, a natural description and code interactive embedding module, and a LGATCS model code recommendation module; The code structure component semantic reorganization module is used to learn the word embedding of the code structure components through the self-attention neural network and integrate the unified representation of the structure components; this module performs the following steps: S1. Learn word embedding representations for the structural components of the code and integrate the unified representation of the structural components through a self-attention neural network; The embedding representation module of natural description is used to learn the word embedding representation of natural description and learn the unified representation of natural description through the self-attention neural network; this module performs the following steps: S2 learns word embedding representations for natural descriptions and learns a unified representation of natural descriptions through a self-attention neural network; The natural description and code interaction embedding module is used to establish the semantic relationship between the natural description and code structure components; this module performs the following steps: S3 uses a self-attention neural network to interactively learn the natural description-code structure component pairs and obtain the interactive representation vectors of the two. The LGATCS model code recommendation module uses the LGATCS model based on the self-attention neural network to give corresponding code recommendations for the user's query; this module performs the following steps: S4, performing an average pooling operation on the interactive representation vectors of the natural description and the code structure component and calculating the cosine similarity between the two for matching; S5. Construct a code search model LGATCS based on the integrated structural component of the self-attention neural network and construct a hinge loss function; S6. Train the hyperparameters of the LGATCS model and recommend codes to user queries based on the obtained model hyperparameter file; S7. Experimental evaluation of the performance of the code search model LGATCS.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed, the code search method based on the focused self-attention integrated structural component as described in any one of claims 1 to 5 is implemented.

8. A computer device, characterized in that: include: Memory and processor; A computer program is stored in the memory, and is characterized in that when the processor calls the computer program, the processor executes the code search method based on the focused self-attention integrated structural component as described in any one of claims 1 to 5.

Images