Dual model method for code generation and code annotation based on Transform model

By adopting dual learning and attention constraint methods in the Transformer model, combining the probability correlation between annotation generation and code generation, the problem of difficult balance of annotation generation and code generation performance and long-distance dependence in the existing technology is solved, and efficient Comment and code generation tasks.

CN118917327APending Publication Date: 2024-11-08SHENYANG UNIVERSITY OF TECHNOLOGY
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410949071.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing comment generation and code generation methods are difficult to balance in performance, and the use of RNN results in parallel computing being unable to solve long-distance dependency problems.

Method used

Using a dual model architecture based on Transformer, the model training process is optimized to improve performance through dual learning and attention constraints, combining the probability correlation between annotation generation and code generation.

Benefits of technology

Effectively balance the performance of annotation generation and code generation tasks, solves the long dependency problem, and improves the training efficiency of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917327A_ABST
    Figure CN118917327A_ABST
Patent Text Reader

Abstract

The invention discloses a dual model method for annotation generation and code generation based on a Transform model, which relates to the technical field of machine translation and is characterized by comprising the following steps of: (1) data processing: converting an input code and an annotation document into a matrix with a fixed size; (2) inputting expression; (3) setting hyper-parameters, and weighting the importance of the loss function in the training process; (4) pre-training: performing a regression task on the code set X and the code Y by using an LSTM model to obtain a code prior probability P (x) and an annotation prior probability P (y); (5) performing dual-task iterative training; and (6) outputting. The dual model architecture is built by using the Transform model combination, so that the performance between annotation generation and code generation tasks can be well balanced, and the problem of long dependence is solved; on the basis of a Transform model, probability correlation between annotation generation and code generation is adopted as a regularization item, and a constraint based on attention is adopted to ensure the similarity of attention weights of two models in the training process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of machine translation, and more specifically, to a machine translation method based on a Transformer dual model. Background Art

[0002] As the basic part of software development, comment generation and code generation are topics that have attracted much attention in the field of software engineering. More and more researchers are applying neural networks to general comment generation and code generation tasks, which can be roughly divided into two categories: supervised learning-based and pre-training-based.

[0003] In the supervised learning-based approach, it is proposed that many artificial intelligence tasks appear in a dual form, that is, the input and output of one task are exactly the output and input of another task. A dual model is designed based on this correlation and has been successfully applied in fields such as machine translation, sentiment classification, and image recognition. As a further derivative of this dual supervised learning, the researchers designed a dual model based on Attention to train the code annotation and code generation models simultaneously. The symmetry between the tokens in the source code snippet and the tokens in the annotation is measured by the attention weight. In order to further strengthen the probabilistic connection between the models, in addition to using the probabilistic correlation between code annotation and code generation as a regularization term in the loss function, the approximation between the attention weight column vector of code generation and the attention weight row vector of code annotation is added to the loss function as a regularization term to ensure the similarity of the attention weights of the two models during training.

[0004] The other type of pre-training usually performs code generation and code comment generation separately. For example, CodeT5 is a pre-training model based on the Transformer architecture. Code and its corresponding comments are used to learn a better alignment between natural language and programming language, that is, natural language → programming language generation and programming language → natural language generation are regarded as symmetric tasks, and the model is optimized at the same time. However, these methods have not yet effectively utilized the intrinsic connection between the two tasks. Taking CodeT5 as an example, comment generation and code generation are designed and trained separately, but the comment generation task is used to help train the code generation task.

[0005] The shortcomings of the annotation generation and code generation methods in the prior art are:

[0006] The performance cannot be well balanced between code generation and code annotation. For example, the BLEU score of code annotation generation in CodeT5 is not satisfactory, although it significantly improves the performance of existing code generation models.

[0007] Using RNN as encoder and decoder to train these two tasks serially makes it impossible to parallelize the computation to solve the long-distance dependency problem. Summary of the invention

[0008] Purpose of the invention: To overcome these problems, this patent proposes an easy-to-train end-to-end method to improve the tasks of comment generation and code generation. Using Transformer to learn the similarity between the generated comments and code can help improve the performance of the code generation model, that is, the dual model architecture can be used to train these two tasks serially so that the performance between the comment generation and code generation tasks is well balanced.

[0009] Technical solution:

[0010] A dual model method of code generation and code annotation based on a Transformer model is characterized by comprising the following steps:

[0011] (1) Data processing: In the data processing stage, the code documents and annotation documents are first converted into a fixed-size matrix representation, followed by feature extraction and sequence padding to ensure consistent length, a vocabulary and word index mapping are established, and word embedding is applied to convert them into dense vectors to capture semantic relationships. Sparse data is further processed, cleaning operations are performed to remove noise and spelling errors, and part-of-speech tagging feature engineering operations are performed to enrich data representation capabilities.

[0012] (2) Input representation: In the input representation stage, we assume that X represents the set of all code snippets and Y represents the set of all comment snippets. On this basis, we establish the comment generation model P(y|x) and the code generation model P(x|y). Specifically, for a code snippet x containing (x1, x2, x3…x m ), and an annotation segment y containing (y1,y2,y3…y n ); These two models are designed to effectively describe and predict the output under a given input, thereby realizing the conversion relationship between code and comments;

[0013] (3) Setting hyperparameters: In the hyperparameter setting stage, the hyperparameter λ is introduced. dual1 ,λ dual2 ,λ sim1 ,λ sim2 , these hyperparameters weight the importance of different parts of the loss function during model training;

[0014] (4) Pre-training:

[0015] In the pre-training stage of the model, the LSTM model is used to perform regression tasks on the code set X and code Y respectively, in order to obtain the code prior probability P(x) and the annotation prior probability P(y), and obtain the probability estimate for the given code and annotation;

[0016] (5) Dual-task iterative training:

[0017] 5.1 First, get a mini batch containing K pairs from the dataset [x k ,y k ];

[0018] 5.2 Forward propagation calculates joint probability;

[0019] 5.3 Forward propagation calculates the similarity of the two models in the transform encoder-decoder multi-head attention weights;

[0020] 5.4 The back-propagation optimizer optimizes the model weights in sequence;

[0021] 5.5 Output: The model weight parameters obtained by final training.

[0022] Based on the Transformer model, the probabilistic correlation between code annotation and code generation is used as a regularization term, and an attention-based constraint is used to ensure the similarity of the attention weights of the two models during training. Code annotation can be regarded as a text generation task. Given an input code snippet x = (x1, x2, x3…x m ), the purpose is to generate readable natural language y = (y1, y2, y3…y n ), these n words describe the input code snippet x. The system tries to find the best sequence y about x.

[0023]

[0024] In contrast, given a natural language annotation y, code generation is about generating the best possible code snippet x for y:

[0025]

[0026] For both Seq2Seq models, the input of code annotation is the expected result of code generation and vice versa. Therefore, code annotation and code generation are dual tasks of each other.

[0027] The purpose of dual learning is to exploit the duality so that two dual tasks can keep learning from each other until convergence. Given two tasks: the first task takes samples from space X as input and maps them to output space Y, and the other task takes samples from space Y as input and maps them to output space X. Therefore, the first task is P(y|x) and the second task is P(x|y). According to Bayes' theorem, if both models work perfectly, there should be:

[0028] P(x, y) = P(x)P(y|x) = P(y)P(x|y) (3);

[0029] where P(x,y) is the joint probability, P(x) and P(x) are the marginal distributions, and P(x|y) and P(y|x) are the conditional probabilities.

[0030] This gives one of the regularization terms:

[0031] L dual =,P(x)P(y|x;W xy )-P(y)P(x|y;W yx )- 2 .

[0032] In the code annotation model and code generation model, the weight matrix of this alignment relationship can be represented by the transform encoder-decoder multi-head attention weight. This process is abstracted into the following steps and expressed in mathematical formulas.

[0033] 1. Obtaining the weight matrix: W sc =θ xy and W cs =θ yx : These two weight matrices are learned by the encoder and represent the relationship between source code and comments. sc represents the strength of association between each element in the source code and each element in the comment, while W cs Indicates an association in the opposite direction.

[0034] 2. Matrix transposition and expansion: v sc and v cs yes and The column vector form of: Transposing these weight matrices is to be able to compare the correspondence between source code and comments. The transposed matrix is ​​expanded into a column vector so that dimensionality reduction and normalization can be applied later.

[0035] 3. Dimensionality reduction: v s ' c and v c ' sis the column vector after dimensionality reduction: Dimensionality reduction is to reduce the complexity of the data while retaining the most important information. This helps the model learn the potential relationship between source code and comments more effectively.

[0036] 4. Normalization: p sc =softmax(v s ' c ) and p cs =softmax(v c ' s ): Normalization is to convert the reduced-dimensional vector into a probability distribution, so that the similarity between two distributions can be compared using information theory methods;

[0037] 5. Calculate similarity: Use Jensen-Shannon divergence D JS Calculate similarity:

[0038] L sim (x i ,y i θ xy ,θ yx )=D JS (p sc ||p cs )

[0039] Jensen-Shannon divergence is a measure of the difference between two probability distributions, where lower values ​​indicate more similar distributions. In this context, L sim It measures the semantic consistency between source code and annotations.

[0040] In this way, L sim (x i ,y i θ xy ,θ yx ) can be used as a regularization term of the model to help the model better learn the correspondence between source code and comments during training, thereby improving the performance of code annotation and code generation tasks.

[0041] In summary, the present invention has the following beneficial effects:

[0042] The dual model architecture built using the Transformer model combination has a good balance between the performance of comment generation and code generation tasks, and solves the long dependency problem.

[0043] We use the probabilistic correlation between comment generation and code generation as a regularizer based on the Transformer model, and employ an attention-based constraint to ensure the similarity of the attention weights of the two models during training. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 is a flow chart in an embodiment of the present invention;

[0045] Figure 2 It is the overall model framework of the present invention;

[0046] Figure 3 It is the weight matrix constraint of the present invention. DETAILED DESCRIPTION

[0047] To overcome these problems, this patent proposes an easy-to-train end-to-end method to improve the tasks of comment generation and code generation. Using Transformer to learn the similarity between the generated comments and code can help improve the performance of the code generation model, that is, a dual model architecture can be used to train these two tasks serially so that the performance between the comment generation and code generation tasks is well balanced.

[0048] Therefore, we use the Transformer model to form a dual model architecture. Since comment generation and code generation tasks are two tasks with dual form, we use dual learning to explicitly exploit the probabilistic correlation between comment generation and code generation, and we also adopt a new constraint on the attention mechanism to enforce the duality.

[0049] The above technical objectives of the present invention are achieved through the following technical solutions: A method based on the Transformer model in the dual tasks of comment generation and code generation, comprising the following steps:

[0050] 1.1 Data processing:

[0051] In the data processing stage, the code documents and comment documents are first converted into fixed-size matrix representations. Feature extraction and sequence padding are then performed to ensure consistent lengths, a vocabulary and word index mapping are established, and word embedding is applied to convert them into dense vectors to capture semantic relationships. Sparse data is further processed, cleaning operations are performed to remove noise and spelling errors, and part-of-speech tagging feature engineering operations are performed to enrich data representation capabilities. These steps are designed to effectively prepare raw text data and lay a solid foundation for model training and application.

[0052] 1.2 Input representation:

[0053] In the input representation stage, we assume that X represents the set of all code snippets, and Y represents the set of all comment snippets. On this basis, we establish the comment generation model P(y|x) and the code generation model is P(x|y). Specifically, for a code snippet x containing (x1, x2, x3…x m ), and an annotation segment y containing (y1,y2,y3…yn ). These two models are designed to effectively describe and predict the output under a given input, thereby achieving a conversion relationship between code and comments.

[0054] 1.3 Setting hyper parameters:

[0055] In the stage of setting hyperparameters, the hyperparameter λ is introduced dual1 ,λ dual2 ,λ sim1 ,λ sim2 These hyperparameters play a key role in the model training process. They weight the importance of different parts of the loss function and help optimize the model training effect. By setting these hyperparameters reasonably, the learning behavior of the model can be controlled to a certain extent, thereby obtaining the best model that is more suitable for a specific task.

[0056] 1.4 Pre-training:

[0057] In the pre-training stage of the model, the LSTM model is used to perform regression tasks on the code set X and code Y respectively, with the aim of obtaining the code prior probability P(x) and the comment prior probability P(y). Through this process, we can obtain probability estimates for given codes and comments, which provides important prior information and basis for subsequent model training, thus helping to better guide the model's decision-making process when generating comments and codes.

[0058] 1.5 Dual-task iterative training:

[0059] (1) First, obtain a mini-batch containing K pairs from the dataset [x k ,y k ].

[0060] (2) The code annotation generation model and the code generation model are serially forward propagated to calculate the joint probability, i.e., Ldual = [P(x)P(y|x;θxy)-P(y)P(x|y;θyx)]2.

[0061] (3) During the forward propagation process, the similarity of the two models in the transform encoder-decoder multi-head attention weights is calculated, i.e., L sim (x i ,y i θ xy ,θ yx ).

[0062] (4) Construct code comments to generate loss function:

[0063]

[0064] (5) Construct code generation model loss function:

[0065]

[0066] (6) The back-propagation optimizer optimizes the model weights in sequence.

[0067] 1.6 Output:

[0068] The model parameters obtained after final training.

[0069] The following is combined with Figure 1-3 The present invention is described in further detail.

[0070] 1.1 Data processing:

[0071] In the data processing stage, the code documents and comment documents are first converted into fixed-size matrix representations. Feature extraction and sequence padding are then performed to ensure consistent lengths, a vocabulary and word index mapping are established, and word embedding is applied to convert them into dense vectors to capture semantic relationships. Sparse data is further processed, cleaning operations are performed to remove noise and spelling errors, and part-of-speech tagging feature engineering operations are performed to enrich data representation capabilities. These steps are designed to effectively prepare raw text data and lay a solid foundation for model training and application. Each step is abstracted into a mathematical operation. The following is a formula representation of each step:

[0072] (1) Matrix representation: Let D code and D sum Represent code documents and comment documents respectively. Convert them into a fixed-size matrix representation M code and M sum :

[0073] M code =f matrix (D code ,size)

[0074] M sum =f matrix (D sum ,size)

[0075] where f matrix is a function that converts text data into a matrix, where size is the fixed size of the matrix. (This is the same as)

[0076] (2) Feature extraction and sequence filling: Feature extraction and sequence filling are performed on the matrix to ensure consistent length:

[0077] M′ code =f extract (M code )

[0078] M′ sum =f extract (Msum )

[0079] M″ code =f pad (M′ code ,max_length)

[0080] M″ sum =f pad (M′ sum ,max_length)

[0081] where f extract is the feature extraction function, f pad is the sequence filling function, max_length is the maximum length of the sequence.

[0082] (3) Build vocabulary and word embedding: Build vocabulary V and word index mapping I, and apply word embedding to convert into dense vector V embed :

[0083] ″″

[0084] V,I=f vocab (M code ,M sum )

[0085] V embed =f embed (V,dim)

[0086] where f vocab is the function that builds the vocabulary, f embed is the word embedding function, and dim is the dimension of the embedding vector.

[0087] (4) Data cleaning and sparse data processing: Data cleaning and sparse data processing:

[0088] M″′ code =f clean (M″ code )

[0089] M″′ sum =f clean (M″ sum )

[0090] M″″ code =f sparsc (M″′ code )

[0091] M″″ sum =f sparsc (M″′ sum )

[0092] where f cleanis the data cleaning function, f sparsc It is a sparse data processing function.

[0093] (5) Part-of-speech tagging: Tag the data with part-of-speech:

[0094] ″″″″′

[0095] M code =f pos (M code )

[0096] ″″″″′

[0097] M sum =f pos (M sum )

[0098] where f pos is the part-of-speech tagging function.

[0099] These steps together constitute the data processing stage, which provides

[0100] ″″″″″

[0101] Prepared data M code and M sum ;

[0102] 1.2 Input representation:

[0103] In the input representation stage, we assume that X represents the set of all code snippets, and Y represents the set of all comment snippets. On this basis, we establish the comment generation model P(y|x) and the code generation model is P(x|y). Specifically, for a code snippet x containing (x1, x2, x3…x m ), and an annotation segment y containing (y1,y2,y3…y n ). These two models are designed to effectively describe and predict the output given an input, thereby achieving a conversion relationship between code and comments. The following is a general table of this process:

[0104] 1.2.1. Sequence representation: Code snippet x and comment snippet y can be represented as:

[0105] x=,x1,x2,x3,…,x m -

[0106] y=,y1,y2,y3,…,y n -

[0107] 1.2.2. Index Mapping and Word Embeddings: Suppose we have a vocabulary V and an index mapping I that maps words to indices in the vocabulary. The word embedding matrix E maps these indices to dense vectors:

[0108] I(x i )→i x

[0109] I(y i )→j y

[0110] E,i x -→v x

[0111] E,j y -→v y

[0112] where v x and v y is x i and i The embedding vector of .

[0113] 1.2.3. Sequence padding: If m≠n, the sequence needs to be padded or truncated to the same length L:

[0114]

[0115] Among them, pad * is the fill element (usually a zero vector).

[0116] 1.2.4. Model input: Finally, the processed sequences x′ and y′ are used as input to the model:

[0117] input=(x′,y′)

[0118] 1.3 Setting hyper parameters:

[0119] In the stage of setting hyperparameters, the hyperparameter λ is introduced dual1 ,λ dual2 ,λ sim1 ,λ sim2 These hyperparameters play a key role in the model training process. They weight the importance of different parts of the loss function and help optimize the model training effect. By setting these hyperparameters reasonably, the learning behavior of the model can be controlled to a certain extent, thereby obtaining the best model that is more suitable for a specific task.

[0120] 1.4 Pre-training:

[0121] In the pre-training stage of the model, the LSTM model is used to perform regression tasks on the code set X and code Y respectively, with the goal of obtaining the code prior probability P(x) and the comment prior probability P(y). Through this process, we can obtain probability estimates for given codes and comments, which provides important prior information and basis for subsequent model training, thus helping to better guide the model's decision-making process when generating comments and codes. This process can be carried out through the following steps:

[0122] 1.4.1. Data preparation: Set the code set X = *x1,x2,…x k + and annotation set Y = *y1,y2,…y l +Process them separately and convert them into a format suitable for LSTM input.

[0123] 1.4.2. LSTM model training: For a code set X, train an LSTM model to predict the given current code snippet x i The next code snippet of x i+1 Similarly, for the annotation set Y, another LSTM model is trained to predict given the current annotation segment y j The next comment segment y j+1 .

[0124] 1.4.3. Regression task: In the LSTM model, the goal of the regression task is to minimize the difference between the predicted sequence and the actual sequence. This can be achieved by using the mean square error (MSE) or similar loss functions.

[0125] 1.4.4. Learning prior probabilities: Through training, the LSTM model can learn the sequence distribution of code and comments, thereby obtaining estimates of the code prior probability P(x) and the comment prior probability P(y)

[0126] 1.4.5. Utilization of prior information: The learned prior probabilities can be used as prior information for subsequent model training, helping the model make more reasonable decisions when generating comments and codes.

[0127] 1.5 Dual-task iterative training:

[0128] (1) First, obtain a mini-batch containing K pairs from the dataset [x k ,y k ].

[0129] (2) The code annotation generation model and the code generation model are serially forward propagated to calculate the joint probability, i.e., Ldual = [P(x)P(y|x;θxy)-P(y)P(x|y;θyx)]2.

[0130] (3) In the forward propagation process, the similarity of the two models in the transform encoder-decoder multi-head attention weights is calculated, i.e., Lsim(x i ,y i θ xy ,θ yx ).

[0131] (4) Construct code comments to generate loss function:

[0132]

[0133] (5) Construct code generation model loss function:

[0134]

[0135] (6) The back-propagation optimizer optimizes the model weights in sequence.

[0136] 1.6 Output:

[0137] The model parameters obtained after final training.

[0138] The following is a verification experiment of this embodiment:

[0139] To evaluate the proposed method, we conducted experiments on Java and Python projects from GitHub. The statistics of these datasets are shown in Table 1.

[0140] Table 1.Details of the datasets

[0141]

[0142] The implementation details are as follows:

[0143] In the dual-model framework, two models are used: the comment generation model and the code generation model. Both models adopt the Transformer architecture. In the dataset, the maximum length of both the code and the comment is set to 64. To optimize these two models, we use the RMSprop optimizer with an initial learning rate of 0.0005. The batch size is set to 64 and the loss rate is set to 0.1. In addition, the initialization of the comment generation model and the code generation model uses a warm-up model optimized by RMSprop with an initial learning rate of 0.002. Warming up involves pre-training the comment generation model and the code generation model separately, and then training them jointly under dual constraints to speed up the convergence process.

[0144] For the dual learning process, we found that using SGD with an initial learning rate of 0.0005 was appropriate. We used a learning rate scheduler that reduces the learning rate to its original value times 0.99 after one epoch. The regularization parameter λ dual1 and λ dual2 are all set to 0.03, and λ sim1 and λ sim2 Both are set to 0.01. The feed-forward network has a dimension of 64, with 2 attention heads and 4 layers. The embedding size and hidden layer dimension are both set to 64.

[0145] In our approach, to compute the marginal probability distribution of each comment and code, we used an LSTM with three layers of LSTM. The hidden state dimension used in these LSTM layers was 300. To process the data efficiently, we set the batch size to 40 and the loss rate to 0.3. To alleviate the problem of gradient explosion, we applied gradient clipping, which helps prevent the gradient from being too large during training. This ensures the stability and effective optimization of the model. To build the vocabulary, we considered all words with a frequency of at least 3. This approach helps capture meaningful and informative words while filtering out less important words. Regarding the choice of optimizer, we chose the Adam optimizer, which performs well in various deep learning tasks. We set the initial learning rate to 0.002 to enable the model to update its parameters efficiently during training. The results are shown in Tables 2 and 3.

[0146] Table 2.Performance comparison of code summary

[0147]

[0148] Table 3.Performance comparison of code generation

[0149]

[0150] This paper proposes a novel approach to formulate dual code annotation and code generation tasks in a transformer-based framework. We develop a joint learning framework to train both code annotation and code generation models simultaneously. To enforce the interdependence between these two tasks during training, we propose the exploitation of dual constraints. Specifically, in addition to the constraint applied to probability, we introduce a constraint that exploits the inherent properties of the similarity mechanism. To evaluate the effectiveness of our proposed model, we conduct extensive experiments on Java and Python datasets. Experimental results show that both code annotation and code generation models achieve superior performance, surpassing the state-of-the-art methods on existing datasets.

[0151] This specific embodiment is merely an explanation of the present invention and is not a limitation of the present invention. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed. However, as long as they are within the scope of the claims of the present invention, they are protected by the patent law.

Claims

1. A dual model method of code generation and code annotation based on Transformer model, characterized by: The following steps are involved: (1) Data processing: In the data processing stage, the code documents and annotation documents are first converted into a fixed-size matrix representation, followed by feature extraction and sequence padding to ensure consistent length, a vocabulary and word index mapping are established, and word embedding is applied to convert them into dense vectors to capture semantic relationships. Sparse data is further processed, cleaning operations are performed to remove noise and spelling errors, and part-of-speech tagging feature engineering operations are performed to enrich data representation capabilities. (2) Input representation: In the input representation stage, we assume that X represents the set of all code snippets and Y represents the set of all comment snippets. On this basis, we establish the comment generation model P(y|x) and the code generation model P(x|y). Specifically, for a code snippet x containing (x1, x2, x3…x m ), and an annotation segment y containing (y1,y2,y3....y n ); These two models are designed to effectively describe and predict the output under a given input, thereby realizing the conversion relationship between code and comments; (3) Setting hyperparameters: In the hyperparameter setting stage, the hyperparameter λ is introduced. dual1 ,λ dual2 ,λ sim1 ,λ sim2 , these hyperparameters weight the importance of different parts of the loss function during model training; (4) Pre-training: In the pre-training stage of the model, the LSTM model is used to perform regression tasks on the code set X and code Y respectively, in order to obtain the code prior probability P(x) and the annotation prior probability P(y), and obtain the probability estimate for the given code and annotation; (5) Dual-task iterative training: 5.1 First, get a mini batch containing K pairs from the dataset [x k ,y k ]; 5.2 Forward propagation calculates joint probability; 5.3 Forward propagation calculates the similarity of the two models in the transform encoder-decoder multi-head attention weights; 5.4 The back-propagation optimizer optimizes the model weights in sequence; 5.5 Output: The model weight parameters obtained by final training.

2. The dual model method of code generation and code annotation based on the Transformer model according to claim 1, characterized in that: In step (1), (1) Matrix representation: Let D code and D sum Represent code documents and comment documents respectively. Convert them into a fixed-size matrix representation M code and M sum : M code =f matrix (D code ,size) M sum =f matrix (D sum ,size) where f matrix is a function that converts text data into a matrix. size is the fixed size of the matrix and obtains the matrix representation M of the source data. sum and M code ; (2) Feature extraction and sequence filling: Feature extraction and sequence filling are performed on the matrix to ensure consistent length: M′ code =f extract (M code ) M′ sum =f extract (M sum ) M″ code =f pad (M′ code ,max_length) M″ sum =f pad (M′ sum ,max_length) where f extract is the feature extraction function, f pad is the sequence filling function, max_length is the maximum length of the sequence, and after two feature extractions and two sequence fillings, the new matrix representation M″ is obtained sum and M″ code ; (3) Build vocabulary and word embedding: Build vocabulary V and word index mapping I, and apply word embedding to convert into dense vector V embed : V,I=f vocab (M″ code ,M″ sum ) In embed =f embed (V,smoke) where f vocab is the function that builds the vocabulary, f embed is the word embedding function, dim is the dimension of the embedding vector; (4) Data cleaning and sparse data processing: Data cleaning and sparse data processing: M″′ code =f clean (M″ code ) M″′ sum =f clean (M″ sum ) M″″ code =f sparsc (M″′ code ) M″″ sum =f sparsc (M″′ sum ) where f clean is the data cleaning function, f sparsc is a sparse data processing function, respectively for M″ sum and M″ code After data cleaning and sparse data processing, a new matrix representation is obtained again; (5) Part-of-speech tagging: Tag the data with part-of-speech: M″″′ code =f pos (M″″ code ) M″″′ sum =f pos (M″″ sum ) where f pos is the part-of-speech tagging function; These steps together constitute the data processing stage, which provides prepared data M″″′ for model training and application code and M″″′ sum .

3. The dual model method of code generation and code annotation based on the Transformer model according to claim 1, characterized in that: Step (5) 5.3 In model training, in the code annotation model and the code generation model, the weight matrix of this alignment relationship is represented by the transform encoder-decoder attention weight; This process is abstracted into the following steps and expressed in mathematical formulas: (1) Obtaining the weight matrix: θ xy and θ yx : These two weight matrices are learned through transform encoder-decoder multi-head attention, representing the relationship between source code and comments; θ xy represents the strength of association between each element in the source code and each element in the comment, and θ yx Indicates association in the opposite direction; (2). Matrix transposition and expansion: v sc and v cs They are and θ yx Column vector form: Transpose θ xy The purpose of the matrix is ​​to obtain the strength of association between each element in the comment and each element in the source code; the purpose of expanding the transposed matrix into a column vector is to be able to apply dimensionality reduction and normalization processing later; (3) Dimensionality reduction: v′ sc and v′ cs is the column vector after dimensionality reduction: Dimensionality reduction is to reduce the complexity of data while retaining the most important information; (4) Normalization: p sc =softmax(v′ sc ) and p cs =softmax(v′ cs ): v′ sc and v′ cs Normalize to get p sc and p cs The purpose is to convert the reduced vector into a probability distribution, so that the similarity between the two distributions can be compared using information theory methods; (5) Calculate similarity: Use Jensen-Shannon divergence D JS Calculate similarity: L sim (x i ,y i ;θ xy ,i yx )=D JS (p sc ||p cs ); D JS It is a way to measure the difference between two probability distributions. The lower its value, the more similar the two distributions are. i ,y i represents the ith code element and the ith comment element; in this context, L sim It measures the semantic consistency between source code and annotations.

4. The dual model method of code generation and code annotation based on Transformer model according to claim 1, characterized in that: In step (5) 5.1 model training, the probability correlation between code annotation and code generation is used as a regularization term based on the Transformer model; code annotation is regarded as a text generation task. Given an input code snippet x = (x1, x2, x3…x m ), the purpose is to generate readable natural language y = (y1, y2, y3…y n ), these n words describe the input code snippet x; the system tries to find the optimal sequence y about x; In contrast, given a natural language annotation y, code generation is about generating the best possible code snippet x for y: For both Seq2Seq models, the input to code annotation is the expected result of code generation, and vice versa; Therefore, code annotation and code generation do double duty with each other.

5. The dual model method of code generation and code annotation based on Transformer model according to claim 4, characterized in that: Based on the Transformer model, the probability correlation between code annotation and code generation is used as a regularization term. Its purpose is to utilize duality so that the two dual tasks can keep learning from each other until convergence. Given two tasks: the first task takes samples from space X as input and maps them to output space Y, and the other task takes samples from space Y as input and maps them to output space X. Therefore, the first task is P(y|x) and the second task is P(x|y). According to Bayes' theorem, if both models run perfectly, there should be: P(x,y)=P(x)P(y|x)=P(y)P(x|y) (3), Where P(x,y) is the joint probability, P(x) and P(x) are marginal distributions, P(x|y) and P(y|x) are conditional probabilities, thus obtaining one of the regularization terms L dual =[P(x)P(y|x; W xy )-P(y)P(x|y;W yx )] 2 Where W xy and W yx They are the overall attention weights of the comment generation model and the code generation model respectively.

6. The dual model method of code generation and code annotation based on Transformer model according to claim 1, characterized in that: In step (5) dual-task iterative training: (1) First, obtain a mini-batch containing K pairs from the dataset [x k ,y k ]; (2) The code comment generation model and the code generation model are serially forward propagated to calculate the joint probability, i.e., L dual =[P(x)P(y|x; W xy )-P(y)P(x|y;W yx )] 2 ; (3) During the forward propagation process, the similarity of the two models in the transform encoder-decoder attention weights is calculated, i.e., L sim (x i ,y i θ xy ,θ yx ); (4) Construct code comments to generate loss function: Where K is the batch size, L xy is the cross entropy loss function of the transform, f xy (x i ; W xy ) is the i-th annotation output by the model; (5) Construct code generation model loss function: Where L yx is the cross entropy loss function of the transform, f yx (y i ; W yx ) is the i-th code output by the model; (6) The back-propagation optimizer optimizes the model weights in sequence.

Citation Information

Cited By

  • Enhanced dual-learning-driven evolutionary optimization method and system for aircraft assembly scheduling

    CN119717732A