Software community warehouse mining method and device based on self-attention interactive network

Abstract

The present invention provides a method and device for mining a software community warehouse based on a self-attention interactive network. Input a code snippet and its context and query description into a software warehouse mining model to obtain a prediction result. The software warehouse mining model includes: a self-attention network layer extracts a feature matrix of the code snippet, the code snippet context and the query description after word embedding processing; an interactive layer obtains the common attention representation of the code snippet and the query description respectively; an information fusion layer obtains the code snippet context, the fusion information of the code snippet and the query description; and a label prediction layer predicts whether the code snippet is an independent solution corresponding to the query description. The use of a self-attention word embedding network has better execution efficiency. The semantic relevance of the code snippet and the query description is fully mined through the interactive layer, and the code snippet, its context information and the query description are fused. Prediction is performed based on the fused information, which effectively improves the prediction accuracy.

Description

Technical Field

[0001] The present invention relates to the field of software repository mining in software engineering, and in particular to a method and device for mining software community repositories based on a self-attention interactive network. Background Art

[0002] Software developer assistance tools based on natural language processing (NLP) technology have been rapidly developed and have been used for code completion, code summarization, automatic source code documentation, deobfuscation, cross-language porting, code retrieval, and even code synthesis from natural language. These success stories are attributed to two properties of software source code: first, it is highly reproducible and statistically predictable, which enables researchers to expand from source code and natural language models created using hand-crafted rules with a long history to data-driven models that have proven to be flexible and relatively easy to create; second, source code can be reused by software developers in large quantities due to the popularity of open source software and the growth of open access, "big code" repositories (such as GitHub and Stack Overflow communities), which host tens of millions of code repositories and programming-related query descriptions and code answers, respectively, and have massive amounts of software data that are being used to train such models.

[0003] However, the statistical models that power many of these applications are only as useful as the data they are trained on: garbage in, garbage out. For specific classes of applications, such as source code retrieval given a natural query, code summarization for natural language, and code generation, all of which involve mining corresponding semantic relationships between natural language and programming language, it is essential to have access to large amounts of high-quality, parallel software data, where the natural language and source code need to be closely aligned with each other.

[0004] The software community, taking Stack Overflow (S0) as an example, provides a source for creating high-quality software repository datasets. The query descriptions are diverse, most of which have corresponding high-quality code snippets and context descriptions, that is, there is always a corresponding independent code solution for the query.

[0005] A research work proposes a BiV-HNN model for software repository mining of “how to do it” questions in the Python and SQL language domains in S0, which assumes that the answer post for a given question has a series of blocks {S1, C1, S2, ..., S i , C i , S i+1 ,...,S L-1 , C L-1 , S L}, that is, L text blocks Si′s and L-1 code blocks C i′s Interweave them to obtain code snippets that truly independently answer the query description. The model first uses the Bi-GRU (bidirectional gated recurrent unit) network to uniformly convert each code block (code snippet and its text context) and query description into a high-dimensional vector space, then uses a fully connected network to concatenate and fuse the code content and query description, and finally uses the Bi-GRU again to fuse the context description embedding of the code block and the fused embedding of the code and query description to obtain the final representation of the comprehensive information of the code snippet to determine whether the code snippet is an independent solution to the query description.

[0006] In actual implementation, the inventors of the present application found that the BiV-HNN model relies on GRU (Gated Recurrent Unit). Inside the GRU, the calculation of the next time step depends on the output of the previous time step. Generally speaking, the flat sequence content of the code is relatively long and belongs to a chapter-level document. The GRU is prone to forgetting historical coding information in a longer text sequence. And from the perspective of computational efficiency, this mode of embedding the code sequentially according to the time step makes it impossible for the GRU to be implemented and calculated in a GPU graphics card. In addition, the BiV-HNN model uses two independent Bi-GRUs to represent the code snippet and the natural language description respectively, ignoring the semantic relationship between the code snippet and its description, which makes the mining performance of the BiV-HNN model insufficient. Summary of the invention

[0007] The present invention aims to solve the shortcomings of the mining performance of the existing BiV-HNN model and provide a software community warehouse mining method and device based on a self-attention interactive network.

[0008] In order to achieve the above-mentioned purpose of the present invention, according to the first aspect of the present invention, the present invention provides a software community warehouse mining method based on a self-attention interactive network, the obtained code snippet, code snippet context and query description are input into a trained software warehouse mining model, the software warehouse mining model outputs a prediction result of whether the code snippet is an independent solution corresponding to the query description, and the software warehouse mining model includes: a word embedding layer, which performs word embedding processing on the code snippet, code snippet context and query description respectively; a self-attention network layer, which extracts the feature matrices of the code snippet, code snippet context and query description after word embedding processing based on the self-attention mechanism; a code snippet and query description interactive layer, which detects the semantic relevance between the code snippet and the query description to obtain the common attention representation of the code snippet and the common attention representation of the query description; an information fusion layer, which fuses the pooling result after average pooling of the feature matrix of the code snippet context, the common attention representation of the code snippet and the common attention representation of the query description through Bi-GRU to obtain fusion information; a label prediction layer, which predicts whether the code snippet is an independent solution corresponding to the query description based on the fusion information.

[0009] In order to achieve the above-mentioned purpose of the present invention, based on the same inventive concept, the second aspect of the present invention provides a software community warehouse mining device based on a self-attention interactive network, including: an input module, used to input code snippets, code snippet contexts and query descriptions into the software warehouse mining module; the software warehouse mining module outputs a prediction result of whether the code snippet is an independent solution corresponding to the query description, and the software warehouse mining module includes: a word embedding module, which performs word embedding processing on the code snippet, the code snippet context and the query description respectively; a self-attention network module, which extracts the feature matrices of the code snippet, the code snippet context and the query description after word embedding processing based on the self-attention mechanism; a code snippet and query description interactive module, which detects the semantic relevance between the code snippet and the query description to obtain the common attention representation of the code snippet and the common attention representation of the query description; an information fusion module, which fuses the pooling result after average pooling of the feature matrix of the code snippet context, the common attention representation of the code snippet and the common attention representation of the query description through a Bi-GRU bidirectional gated recurrent unit to obtain fusion information; a label prediction module, which predicts whether the code snippet is an independent solution corresponding to the query description based on the fusion information.

[0010] In order to achieve the above-mentioned purpose of the present invention, based on the same inventive concept, the third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the method described in the first aspect of the present invention when executed.

[0011] In order to achieve the above-mentioned purpose of the present invention, based on the same inventive concept, the fourth aspect of the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in the first aspect of the present invention is implemented.

[0012] The above technical solution: Compared with the GRU model that can only extract limited context information for each element in the sequence and cannot realize parallel computing in the GPU, the present invention adopts a self-attention word embedding network with better execution efficiency, and can model the code snippet and its text context and query description, which can better capture the context information of each element in the sequence, realize accurate mapping from natural language to programming language, so as to improve the performance of software tasks such as code generation, code retrieval and code summarization across natural language and programming language; Different from the model that represents code snippets and query descriptions in isolation in previous work, the present invention adopts a network based on Bi-GRU bidirectional interaction to model the attention signals in the code snippets in the query description and the corresponding independent solution sufficiently; At the same time, the present invention uses an attention mechanism to train and learn the importance weights of code snippets and query descriptions. This novel weighted model can give different attention to the query description and code snippet terms in their respective directions, thereby generating different attention to the query description; The semantic relevance of the code snippet and the query description is fully mined through the interaction layer, and the code snippet, its context information and the query description are fused, and prediction is made based on the fused information, which effectively improves the prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a schematic diagram of the overall structure of the software warehouse mining model in Example 1 of the present invention;

[0014] Figure 2 is a detailed structural diagram of the software warehouse mining model in Example 1 of the present invention;

[0015] Figure 3 is a schematic diagram of the structure of a software community warehouse mining device in Embodiment 2 of the present invention;

[0016] Figure 4 is a block diagram of a computer-readable storage medium in Embodiment 3 of the present invention;

[0017] Figure 5 It is a structural diagram of the computer device in Example 4 of the present invention. DETAILED DESCRIPTION

[0018] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be understood as limiting the present invention.

[0019] In the description of the present invention, it is necessary to understand that the terms "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0020] In the description of the present invention, unless otherwise specified and limited, it should be noted that the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a mechanical connection or an electrical connection, or it can be the internal connection between two components. It can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to the specific circumstances.

[0021] The inventors of this application have found through extensive research and practice that:

[0022] The BiV-HNN model, a method for mining S0 problem codes in the prior art, uses Bi-GRU as the underlying architecture, in which GRU can only extract limited context information for each element in the sequence. From the perspective of efficiency, this mode makes it impossible for GRU to implement parallel computing in GPU (Graphics Processing Units). Although CNN can better implement parallel computing, it is not suitable for encoding sequences. The reason is that for each element, due to the limited size of the convolution kernel, CNN only extracts local semantic information for it. Therefore, if you want to extract global semantic information through CNN, you need to stack CNN multiple times, which makes the model complicated. As for the self-attention network, it is designed based on the self-attention mechanism. Specifically, when representing an element in a sequence, the self-attention mechanism makes this element pay attention to all elements in the sequence through a dot product operation, and then represents this element through a weighted average operation, which shows that the self-attention mechanism does not need to consider the distance factor. Therefore, the self-attention network can capture the context information of each element in the sequence. Considering the execution efficiency, the sequential operation of the self-attention mechanism is O(1), while GRU requires O(n) sequential operations. In terms of computational complexity, the self-attention layer is faster than the GRU layer when the sequence length n is less than the word representation dimension d. In addition, the self-attention network can achieve parallel computing on the GPU because the core of the self-attention network is the dot product operation. Therefore, the self-attention network has achieved state-of-the-art performance in machine translation tasks that require high semantic information, which shows that the self-attention mechanism can capture contextual information better than GRU and CNN.

[0023] In addition, the modeling of code snippets and query descriptions in BiV-HNN reuses some existing dual-mode neural network models in natural language processing, rather than designing interactive modeling specifically for query descriptions and source code. They are unable to fuse the interactive information between the code and its query description, thereby effectively capturing deeper semantic matching signals in heterogeneous languages. At the same time, different words in the query description pay different attention to different tokens in the code snippet, which is usually called an attention mechanism. It turns out that in most natural language processing and programming language understanding tasks, such as machine translation and automatic code summarization research, the attention mechanism in code mining can capture more information about different parts of the code snippet in the independent solution that the query is concerned about.

[0024] In general, existing work on mining community software repositories in S0 is affected by the incomplete information of capturing code snippets and their contexts and query description sequences when predicting whether a code snippet is an independent solution to a query description, and ignores the semantic association between the code snippet and the query description.

[0025] Based on the above considerations, the present invention proposes a method for mining software community repositories based on a self-attention interactive network, which is a new network model for mining software repositories in S0 community instances. The present invention first uses a self-attention network to capture the feature matrix of code blocks and query descriptions; then learns the alignment attention matrix of query descriptions and code snippets to obtain attention projection vectors in different directions, and finally fuses the information of the code snippets and their context and query description to predict whether the code snippets are independent solutions to the query description. The present invention is empirically evaluated based on two large-scale manually annotated Python and SQL code libraries previously collected from the S0 community. The results show that the present invention can effectively predict whether a code snippet is an independent code solution to the query description. The inventive concept of the present invention is described in detail below through four embodiments.

[0026] Example 1

[0027] This embodiment discloses a method for mining a software community repository based on a self-attention interactive network. The obtained code snippet, code snippet context, and query description are input into a trained software repository mining model. The software repository mining model outputs a prediction result of whether the code snippet is an independent solution corresponding to the query description. The overall structure of the software repository mining model is as follows: Figure 1 As shown in the figure, it specifically includes word embedding layer, self-attention network layer, code snippet and query description interaction layer, information fusion layer and label prediction layer.

[0028] (1) Word embedding layer: word embedding is performed on the code snippet, code snippet context, and query description respectively.

[0029] In a given code sequence and its text context sequence and the corresponding query description sequence Q = q1, ..., q |Q| When, |C i | indicates a code snippet |C i |Q| represents the total number of words in the query description, |S i | indicates the total number of words in the context of the code snippet, |S i+1 | represents the total number of words in the following code snippet. First, use the existing word embedding dictionary to query the corresponding word embedding of each word in the sequence to construct its corresponding sequence encoding vector. The query formula is as follows:

[0030]

[0031]

[0032]

[0033]

[0034] in, is the matrix of pre-trained code word embedding, is the pre-trained natural language text word embedding matrix, are the kth word in the code snippet The embedding vector of the lth word q in the query description l The embedding vector of the mth word in the code snippet The embedding vector of the nth word in the following code snippet The embedding vector V C and V T are the fixed-size vocabularies for code and natural language respectively, and d is the dimension of word embeddings.

[0035] (2) The self-attention network layer extracts the feature matrices of the code snippet, code snippet context, and query description after word embedding based on the self-attention mechanism to capture the historical and future information of each word. The self-attention network layer specifically performs:

[0036] Step S21, respectively query weight W Q ∈R d×d , key weight W K ∈R d×d , value weight W V ∈R d×d The code snippet / code snippet context / query description after word embedding processing is converted into corresponding query vector, key vector and value vector.

[0037] Specifically, for the code snippet after word embedding After constructing the word embedding for each word, each word has the same representation dimension d, and the weight W is queried through three different weights Q ∈R d×d , key weight W K ∈R d×d , value weight W V ∈R d×d C i e Converted to the query vector corresponding to the code snippet Key Vector Sum value vector Specific:

[0038] According to the above process, the query description sequence after word embedding and the context of the code snippet are converted into the corresponding query vector, key vector and value vector, which will not be repeated here. e =q1e , ..., q |Q| e Convert to query vector Q Q ∈R |Q|×d , key vector K Q ∈R |Q|×d , value vector V Q ∈R |Q|×d . For the context sequence of the code snippet after word embedding and S i e Convert to the corresponding query vector Key Vector Sum value vector S i+1 e Convert to the corresponding query vector Key Vector Sum value vector

[0039] Step S22, input the query vector, key vector, and value vector corresponding to the code snippet / code snippet context / query description into the self-attention mechanism to obtain the semantic information of the code snippet / code snippet context / query description. Figure 2 As shown, four self-attention networks are set up, corresponding to the code snippet, the context above the code snippet, the context below the code snippet, and the query description respectively.

[0040] Input the query vector, key vector, and value vector of the code snippet into the corresponding self-attention network to obtain the semantic information of the code snippet

[0041]

[0042] Among them, Att(·) represents the scaled dot product attention model

[0043] SoftMax(·) is a normalization function.

[0044] Input the query vector, key vector, and value vector of the query description into the corresponding self-attention network to obtain the semantic information context of the code snippet Q :

[0045]

[0046] Input the query vector, key vector, and value vector of the code snippet into the corresponding self-attention network to obtain the semantic information of the code snippet

[0047]

[0048] Input the query vector, key vector, and value vector of the code snippet into the corresponding self-attention network to obtain the semantic information of the code snippet.

[0049]

[0050] Step S23, inputting the semantic information of the code snippet / code snippet context / query description into a fully connected feedforward network, and the fully connected feedforward network outputs a feature matrix of the code snippet / code snippet context / query description.

[0051] The semantic information of the code snippet, code snippet context, and query description are applied to a fully connected feedforward network to obtain the corresponding feature matrix. The feature matrix of the code snippet / code snippet context / query description is:

[0052] v D =ReLu(context D ·W1+b1)·W2+b2;

[0053] Among them, sequence D = C i or S i or S i+1 or Q, C i Indicates a code snippet, S i Indicates the code snippet above, S i+1 represents the following code snippet, Q represents the query description; ReLu(·) represents the ReLu activation function; context D Represents the semantic information of sequence D; context D =Att(Q D ·K D )·V D , Att(·) represents the scaled dot-product attention model, SoftMax(·) is the normalization function, Q D represents the query vector of sequence D, K D represents the key vector of sequence D, V D represents the value vector of sequence D; d represents the dimension of the word after word embedding; W1, W2, b1 represent the first learnable parameter, the second learnable parameter, and the third learnable parameter in the fully connected feedforward network, respectively. Specifically: when D = C i , the feature matrix of the code snippet is:

[0054]

[0055] When D=S i , the feature matrix above the code snippet is:

[0056]

[0057] When D=S i+1 , the feature matrix below the code snippet is:

[0058]

[0059] When D = Q, the feature matrix of the query description is: Q =ReLu(context Q ·W1+b1).W2+b2.

[0060] Since previous work uses Bi-GRU, a recurrent neural network based on GRU, to learn basic representations by summarizing context information in two directions, compared to Bi-GRU which can only extract limited context information for each element in the sequence, this embodiment uses a self-attention network to extract all context information for each element in the sequence to better capture the historical and future information of each word.

[0061] (3) The code snippet and query description interaction layer detects the semantic relevance between the code snippet and the query description to obtain the co-attention representation of the code snippet and the co-attention representation of the query description. The code snippet and query description interaction layer specifically performs:

[0062] Step S31, calculate the semantic correlation matrix F between the code snippet and the query description:

[0063]

[0064] in, The feature matrix representing the code snippet; v Q represents the feature matrix of the query description; U represents the parameter matrix of the semantic relevance detection neural network, which is obtained through neural network learning; the semantic relevance matrix F can perform collaborative attention observation on the semantic relevance between the code snippet code words and the query description words, and each element F in F k，l (kth row and lth column element) represents the semantic correlation between the two alignment vectors, that is, represents the kth code word, represents the lth query description word. Specifically, the kth row in F represents the semantic association between each word in the query description and the kth code word. The lth column in F represents the semantic association between each word in the code and the lth description word.

[0065] Step S32: Perform an average pooling operation along the rows of the semantic correlation matrix F to obtain the semantic vector of the code snippet. The semantic correlation matrix F is averaged along the columns to obtain the semantic vector g of the query description. Q , specifically:

[0066] Perform average pooling operations along rows and columns on F to obtain semantic vectors for code snippets and query descriptions. and g Q ∈R |Q| . The kth element of represents the code fragment C i The importance score of the kth word in the query description Q. Similarly, g Q The lth element of the query Q represents the relationship between the lth word in the query description Q and its i The importance scores of all words in . Therefore, the average pooling operation we take is as follows:

[0067]

[0068] g i Q =avgpooling[F l，1 , …, F l，|Q| ].

[0069] Semantic vector and g Q The following is obtained:

[0070]

[0071] g Q =[g1 Q , ..., g |Q| Q ].

[0072] Step S33: semantic vector of code snippet Generate the attention vector of the code snippet using the softmax function The semantic vector g of the query description Q Use the softmax function to generate the attention vector a of the query description Q .

[0073] Specifically, the softmax function converts the semantic vector of the code snippet The lth element of Convert to attention ratio The softmax function transforms the semantic vector g described by the query into Q The lth element g l Q Converted to attention ratio a l Q ,

[0074] Step S34, calculating the feature matrix of the code snippet and the attention vector of the code snippet The dot product of the code snippet is used to obtain the common attention representation of the code snippet. Calculate the feature matrix v of the query description Q and the attention vector a of the query description Q The dot product of the query description is used to obtain the common attention representation r Q , r Q ∈R d .

[0075] The interactive layer between code snippets and query descriptions uses a two-way interactive method to extract interactive information. Since previous work ignored the semantic relevance between code snippets and query descriptions, this embodiment introduces an interactive attention mechanism to solve this problem by dynamically aligning the corresponding parts of the code and query vectors. Before the average pooling or maximum pooling operation, the embedding vector output based on the self-attention encoding will be multiplied by the weight after the softmax operation, which is determined by the embedding vector sequence in the self-attention. The interactive embedding method based on the attention mechanism is different from the model that represents the code snippet and query description in isolation in previous work. The present invention uses a network based on two-way interaction to model the attention signals in the code snippet in the query description and the corresponding independent solution sufficiently. At the same time, the present invention uses an attention mechanism to train and learn the importance weights of the code and query description. This novel weighted model can give different attention to the query description and code snippet terms in their respective directions.

[0076] (4) Information fusion layer: The fusion information is obtained by fusion of the pooling result after average pooling of the feature matrix of the code snippet context, the common attention representation of the code snippet, and the common attention representation of the query description through Bi-GRU.

[0077] Specifically, we obtain the joint attention representation of the code snippet and the query description and r Q ∈R d , and the feature matrices of the text context of the code snippet are and In order to fuse the code snippet with its textual context and query description, the feature matrix Perform average pooling to obtain the final representation of the text context and The average pooling operation taken is as follows:

[0078]

[0079]

[0080] For the obtained sequence representation Bundle Treat four words as a sequence and use Bi-GRU for fusion. The bidirectional GRU captures the historical and future information of each step from two directions. The forward and backward hidden states of the sequential input vector can be represented as follows:

[0081]

[0082] Finally, the two hidden states and concatenate to form the final hidden state h i :

[0083]

[0084] in, Represents the concatenation of two vectors, let the hidden unit of each unidirectional GRU be numbered u, and the embedding vector based on Bi-GRU is obtained from the input x of length n. t The updated formula is written as H∈R n×2u :

[0085] H=[h1,...h t , ..., h n ];

[0086] According to the above principle, we finally get The fusion information H i ∈R 4×2u .

[0087] (5) The label prediction layer predicts whether the code snippet is an independent solution corresponding to the query description based on the fusion information. The label prediction layer makes predictions according to the following formula:

[0088] y i =softmax(w y H i +b y );

[0089] Among them, y i Represents the prediction result, y i =[y i0 ,y i1 ],y i0 is the label for the code snippet that does not correspond to the query description and is an independent solution. i1 Indicates that the code snippet is a label representing an independent solution corresponding to the query description, H i represents fusion information, w y 、b y denote the first learnable parameter and the second learnable parameter of the label prediction layer respectively.

[0090] In this embodiment, the machine context of a code snippet and a query description are taken as input to predict whether the code snippet is an independent solution to the query description.

[0091] In this embodiment, in order to improve the prediction accuracy of the model, in the model training, the loss function constructed does not simply fit the one hot distribution, but constructs the loss function based on the one hot distribution and the uniform distribution. Preferably, the training process of the software warehouse mining model is: based on the manually annotated data set, a training set and a validation set are obtained, the network parameters of the software warehouse mining model are trained using the training set, and the validation set is iterated a preset number of times until the loss function converges, and the software warehouse mining model training is completed, wherein the loss function is constructed based on the one hot distribution and the uniform distribution, specifically:

[0092]

[0093] Among them, Loss represents loss; ε represents the proportion of uniformly distributed loss function, 0≤ε≤1; the first variable y i0 The label indicating that the code snippet is not an independent solution corresponding to the query description, p i0 Represents y i0 Confidence level, 0≤p i0 ≤1;y i1 Indicates that the code snippet is a label representing an independent solution corresponding to the query description, p i1 Represents y i1 Confidence level, 0≤p i1 ≤1. The idea behind this is that when the prediction result is [y i_1 ,y i_2 ], the prediction target is [1, 0], blindly increasing the ratio so that If it is close to 1, it is no longer the optimal solution, which can alleviate the overconfidence of softmax. If the i-th code snippet is manually annotated as a solution, then p i0 = 0 and p i1 =1; otherwise, p i0 =1 and p i1 =0.

[0094] In this embodiment, preferably, the pre-trained structured word embedding is combined to fuse the structured knowledge provided by the model's external software data in advance, and manually annotated datasets from Python and SQL domains are used.

[0095] In the process of model training, the word embedding obtained by pre-training in the previous work is first loaded into the software warehouse mining model (SACodeMF model). Then, in the initialization of the software warehouse mining model parameters, the maximum length of the code snippet, the context of the code snippet, and the query description are set to 302, 100, and 34 respectively, and the attention dimension is 150. In Python language, the probability of sampling is 0.25 in the word embedding layer, the self-attention network layer, the query description and code snippet interaction layer, and the prediction label layer, and the probability of sampling is 0.4 in the information fusion layer; in SQL language, the probability of sampling is 0.25 in the word embedding layer, and the probability of sampling is 0.5 in the self-attention network layer, the query description and code snippet interaction layer, the information fusion layer, and the prediction label layer. After obtaining the training set, the hyperparameters of the software warehouse mining model (SACodeMF model) are trained and fitted according to the learning rate of 0.001 and the Aadm stochastic gradient descent method until the training is stopped after 150 iterations on the validation set, and the hyperparameter file of the software warehouse mining model (SACodeMF model) is saved.

[0096] In order to illustrate the performance of the software warehouse mining model (SACodeMF model) of the present invention in software warehouse mining in the software community, the model performance test is carried out on the manually labeled test set taking Python and SQL domains as examples according to the trained software warehouse mining model (SACodeMF model), and the corresponding Precision, Recall, F1, Accuracy evaluation indicators can be given, and compared with the existing benchmark model BiV-HNN model to illustrate the performance of the software warehouse mining model (SACodeMF model) of the present invention. The experimental results are shown in Tables 1 and 2. The experimental effect of the present invention is better than the BiV-HNN baseline model. The results show that the method of the present invention, because of the use of the self-attention network and the interactive embedding method, more accurately predicts the code snippet (independent code solution) corresponding to the query description than the previous method. Therefore, given a code block (code snippet and its context) and a query description, the present invention can more accurately predict whether a code snippet is an independent solution to the query description.

[0097] Table 1 Evaluation indicators of different software repository mining models on the Python domain test set

[0098]

[0099] Table 2 Evaluation indicators of different software warehouse mining models on the SQL domain test set

[0100]

[0101] The innovative features of the present invention include:

[0102] 1) Embedding method based on self-attention network: Compared with the GRU model that can only extract limited context information for each element in the sequence and cannot implement parallel computing in the GPU, the present invention adopts a coding embedding network based on self-attention, which has better execution efficiency and can model code snippets and their text context and query descriptions, and better capture the context information of each element in the sequence.

[0103] 2) Interactive embedding method based on attention mechanism. Different from the model that represents code snippets and query descriptions in isolation in previous work, this paper adopts a network based on bidirectional interaction to model the attention signals in the query description and the code snippets in the corresponding independent solutions sufficiently well. At the same time, this paper uses the attention mechanism to train and learn the importance weights of code and query descriptions. This novel weighted model can give different attention to the query description and code snippet terms in their respective directions, thereby generating different attention to the query description.

[0104] In addition, it should be noted that the mining network model for the software warehouse in this embodiment can adopt other neural network architectures, and can be applicable to other software communities besides the S0 community instance, so this embodiment does not make specific limitations and descriptions on this.

[0105] This embodiment can capture the semantics of query descriptions and code snippets and their textual contexts in the software community repository to determine whether the code snippet is an independent code solution corresponding to the program query description. The present invention focuses on software communities, such as Stack Overflow's "How-to-do-it" type questions, which ask how to implement a task, so the answers to these questions are likely to be independent code solutions. In this embodiment, first, for each code snippet and its textual context and query description, a self-attention interactive model SACodeMF is constructed, including a self-attention network layer, a query description and code snippet interaction layer, an information fusion layer, and a label prediction layer, and a loss function for the training network is set, wherein the self-attention network layer is used to capture and analyze each code block (code snippet C i and its textual context S i , S i+1) and query description sequence information, the interactive attention layer is used to capture and encode the interactive information of the code segment and the query description, the information fusion layer fuses the context of the code segment and the information of the code segment and the query description, the label prediction layer is used to predict whether the code segment is an independent solution to the problem, and then the pre-trained structure word embedding is loaded into the software warehouse mining model (self-attention interactive model SACodeMF), and the parameters of the software warehouse mining model are trained on the manually annotated dataset taking Python and SQL languages ​​as examples. Finally, the trained software warehouse mining model is used to classify the many code snippets to be processed to determine whether the code snippet is an independent solution corresponding to the program query description.

[0106] Through this method based on self-attention to capture sequence information and combining it with interactive embedding, the present invention can provide a high-quality software community repository, which contains query and code pairs to achieve accurate mapping from natural language to programming language, so as to improve the performance of software tasks such as code generation, code retrieval and code summarization across natural language and programming language.

[0107] Example 2

[0108] Based on the same inventive concept, this embodiment discloses a software community warehouse mining device based on a self-attention interactive network, such as Figure 3 As shown, it includes: an input module, which is used to input code snippets, code snippet contexts and query descriptions into a software warehouse mining module; the software warehouse mining module outputs a prediction result of whether the code snippet is an independent solution corresponding to the query description, and the software warehouse mining module includes: a word embedding module, which performs word embedding processing on the code snippet, the code snippet context and the query description respectively; a self-attention network module, which extracts the feature matrices of the code snippet, the code snippet context and the query description after word embedding processing based on the self-attention mechanism; a code snippet and query description interaction module, which detects the semantic relevance between the code snippet and the query description to obtain the common attention representation of the code snippet and the common attention representation of the query description; an information fusion module, which fuses the pooling result after average pooling of the feature matrix of the code snippet context, the common attention representation of the code snippet and the common attention representation of the query description through Bi-GRU to obtain fusion information; a label prediction module, which predicts whether the code snippet is an independent solution corresponding to the query description based on the fusion information.

[0109] In this embodiment, preferably, Figure 3As shown, it also includes a model training module, which obtains a training set and a validation set based on a manually annotated data set, uses the training set to train the network parameters of the software warehouse mining model, iterates a preset number of times on the validation set until the loss function converges, and completes the software warehouse mining model training, wherein the loss function is constructed based on onehot distribution and uniform distribution, specifically:

[0110]

[0111] Among them, Loss represents loss; ε represents the proportion of uniform distribution loss function; the first variable y i0 The label indicating that the code snippet is not an independent solution corresponding to the query description, p i0 Represents y i0 Confidence of y i1 Indicates that the code snippet is a label representing an independent solution corresponding to the query description, p i1 Represents y i1 confidence level.

[0112] Since the device introduced in Example 2 of the present invention is a device used to implement the method for mining software community warehouses based on self-attention interactive networks in Example 1 of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in Example 1 of the present invention, so it is not repeated here. All devices used in the method of Example 1 of the present invention belong to the scope of protection of the present invention.

[0113] Example 3

[0114] Based on the same inventive concept, this embodiment discloses a computer-readable storage medium 300, see Figure 4 , on which a computer program 311 is stored, which implements the method in Example 1 when executed.

[0115] Since the computer-readable storage medium 300 introduced in Embodiment 3 of the present invention is the computer-readable storage medium 300 used to implement the method for mining software community warehouses based on self-attention interactive networks in Embodiment 1 of the present invention, based on the method introduced in Embodiment 1 of the present invention, those skilled in the art can understand the specific structure and deformation of the computer-readable storage medium 300, so it is not repeated here. All computer-readable storage media 300 used in the method in Embodiment 1 of the present invention belong to the scope of protection of the present invention.

[0116] Example 4

[0117] Based on the same inventive concept, this embodiment discloses a computer device, see Figure 5, including a memory 401, a processor 402, and a computer program 403 stored in the memory 401 and executable on the processor 402, the processor 402 implementing the method in Example 1 when executing the above program

[0118] Since the computer device introduced in Example 4 of the present invention is a computer device used to implement the method for mining software community warehouses based on self-attention interactive networks in Example 1 of the present invention, based on the method introduced in Example 1 of the present invention, those skilled in the art can understand the specific structure and deformation of the computer device, so it is not repeated here. All computer devices used in the method in Example 1 of the present invention belong to the scope of protection of the present invention.

[0119] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0122] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if these modifications and variations of the embodiments of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A software community warehouse mining method based on self-attention interactive network, characterized by: The acquired code snippet, code snippet context and query description are input into a trained software warehouse mining model, and the software warehouse mining model outputs a prediction result of whether the code snippet is an independent solution corresponding to the query description. The software warehouse mining model includes: The word embedding layer performs word embedding processing on the code snippet, code snippet context, and query description respectively; The self-attention network layer extracts the feature matrices of the code snippet, code snippet context, and query description after word embedding based on the self-attention mechanism; The code snippet and query description interaction layer detects the semantic relevance between the code snippet and the query description to obtain the co-attention representation of the code snippet and the co-attention representation of the query description; The information fusion layer uses Bi-GRU to fuse the pooling result of the feature matrix of the code snippet context, the co-attention representation of the code snippet, and the co-attention representation of the query description to obtain fusion information; The label prediction layer predicts whether the code snippet is an independent solution corresponding to the query description based on the fusion information.

2. The method for mining software community warehouses based on self-attention interactive networks according to claim 1, characterized in that: The self-attention network layer specifically performs: Step S21, converting the code snippet / code snippet context / query description after word embedding processing into a corresponding query vector, key vector and value vector by using query weight, key weight and value weight respectively; Step S22, inputting the query vector, key vector, and value vector corresponding to the code snippet / code snippet context / query description into the self-attention mechanism to obtain semantic information of the code snippet / code snippet context / query description; Step S23, inputting the semantic information of the code snippet / code snippet context / query description into a fully connected feedforward network, and the fully connected feedforward network outputs a feature matrix of the code snippet / code snippet context / query description.

3. The method for mining software community warehouses based on self-attention interactive networks as claimed in claim 2, characterized in that: The feature matrix of code snippet / code snippet context / query description is: v D =ReLu(context D ·W1+b1)·W2+b2, where sequence D=C i or S i or S i+1 or Q, C i Indicates a code snippet, S i Indicates the code snippet above, S i+1 represents the following code snippet, Q represents the query description; ReLu(·) represents the ReLu activation function; context D Represents the semantic information of sequence D; context D =Att(Q D ·K D )·V D , Att(·) represents the scaled dot-product attention model, SoftMax(·) is the normalization function, Q D represents the query vector of sequence D, K D represents the key vector of sequence D, V D Represents the value vector of sequence D; d represents the dimension of the word after word embedding; W1, W2, b1 represent the first learnable parameter, the second learnable parameter, and the third learnable parameter in the fully connected feedforward network, respectively.

4. The method for mining software community warehouses based on self-attention interactive networks according to claim 1, characterized in that: The code snippets and queries describe how the interaction layer performs: Step S31, calculate the semantic correlation matrix F between the code snippet and the query description: in, The feature matrix representing the code snippet; v Q represents the feature matrix of the query description; U represents the parameter matrix of the semantic relevance detection neural network; Step S32: Perform an average pooling operation along the rows of the semantic correlation matrix F to obtain the semantic vector of the code snippet. The semantic correlation matrix F is averaged along the columns to obtain the semantic vector g of the query description. Q ; Step S33: semantic vector of code snippet Generate the attention vector of the code snippet using the softmax function The semantic vector g of the query description Q Use the softmax function to generate the attention vector of the query description Step S34, calculating the feature matrix of the code snippet and the attention vector of the code snippet The dot product of the code snippet is used to obtain the common attention representation of the code snippet. Calculate the feature matrix v of the query description Q and the attention vector a of the query description Q The dot product of the query description is used to obtain the common attention representation r Q .

5. The method for mining software community warehouses based on self-attention interactive networks according to claim 1, characterized in that: The label prediction layer performs prediction according to the following formula: y i =softmax(w y H i +b y ): Among them, y i Represents the prediction result, y i =[y i0 ,y i1 ],y i0 is the label for the code snippet that does not correspond to the query description and is an independent solution. i1 Indicates that the code snippet is a label representing an independent solution corresponding to the query description, H i represents fusion information, w y 、b y They represent the first learnable parameter and the second learnable parameter of the label prediction layer respectively.

6. The method for mining software community warehouses based on a self-attention interactive network according to any one of claims 1 to 5, characterized in that: The training process of the software warehouse mining model is as follows: a training set and a validation set are obtained based on a manually annotated data set, the network parameters of the software warehouse mining model are trained using the training set, and the software warehouse mining model training is completed by iterating a preset number of times on the validation set until the loss function converges, wherein the loss function is constructed based on one hot distribution and uniform distribution, specifically: Among them, Loss represents loss; ε represents the proportion of uniform distribution loss function; the first variable y i0 The label indicating that the code snippet is not an independent solution corresponding to the query description, p i0 Represents y i0 Confidence of y i1 Indicates that the code snippet is a label representing an independent solution corresponding to the query description, p i1 Represents y i1 confidence level.

7. A software community warehouse mining device based on a self-attention interactive network, characterized in that: include: An input module, used for inputting code snippets, code snippet contexts and query descriptions into a software repository mining module; The software warehouse mining module outputs a prediction result of whether the code snippet is an independent solution corresponding to the query description, and the software warehouse mining module includes: The word embedding module performs word embedding processing on the code snippet, code snippet context and query description respectively; The self-attention network module extracts the feature matrices of the code snippet, code snippet context, and query description after word embedding based on the self-attention mechanism; The code snippet and query description interaction module detects the semantic relevance between the code snippet and the query description to obtain the co-attention representation of the code snippet and the co-attention representation of the query description; The information fusion module uses Bi-GRU to fuse the pooling result of the feature matrix of the code snippet context, the co-attention representation of the code snippet, and the co-attention representation of the query description to obtain fusion information; The label prediction module predicts whether the code snippet is an independent solution corresponding to the query description based on the fusion information.

8. The software community warehouse mining device based on self-attention interactive network as claimed in claim 7, characterized in that: The system also includes a model training module, which obtains a training set and a validation set based on a manually annotated data set, uses the training set to train the network parameters of the software warehouse mining model, iterates a preset number of times on the validation set until the loss function converges, and completes the software warehouse mining model training, wherein the loss function is constructed based on one hot distribution and uniform distribution, specifically: Among them, Loss represents loss; ε represents the proportion of uniform distribution loss function; the first variable y i0 The label indicating that the code snippet is not an independent solution corresponding to the query description, p i0 Represents y i0 Confidence of y i1 Indicates that the code snippet is a label representing an independent solution corresponding to the query description, p i1 Represents y i1 confidence level.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed, the method according to any one of claims 1 to 6 is implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

Images