Program information generation model training method and device, electronic equipment and storage medium

Abstract

The invention discloses a program information generation model training method and device and electronic equipment, and belongs to the technical field of computers. The method comprises the steps that an encoder of a training program information generation model is based on a first training data set, and the encoder is used for conducting feature extraction and alignment processing on codes and annotations in input program information pairs to obtain code vectors and text vectors; performing encoding processing on the program information pairs in the first training data set based on an encoder obtained by training to obtain a text vector and a code vector corresponding to each program information pair; a decoder training a program information generation model based on the text vector and the code vector; and constructing a program information generation model based on the encoder and the decoder obtained by training. According to the method, feature extraction is performed by adopting feature alignment of the annotation and the code, so that the code and the annotation feature obtained during feature extraction of the model can be reflected to the same space to a great extent, and the accuracy of generating the code by the model based on the annotation is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a program information generation model training method, a program information generation model training device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Code generation algorithms and annotation generation algorithms based on deep learning are one of the important research directions in the field of artificial intelligence in recent years. The former can generate corresponding codes according to code annotations to improve development efficiency, while the latter can automatically analyze the code and generate accurate and clear annotation text to improve the readability of the code. Common code generation methods and annotation generation methods in the prior art mainly include: generation methods based on deep learning and generation methods based on generative models. However, the above methods have the following defects: the code generation method based on deep learning is based on semantic analysis of the input, and then matches the corresponding code from the code library, and cannot generate new code; and the method of generating code based on the generative model cannot ensure the accuracy of the generated code due to the randomness of the generated code.

[0003] It can be seen that the code generation method in the prior art still needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide a method and device for training a program information generation model, which helps to improve the accuracy of generating program code based on program code description text.

[0005] In a first aspect, an embodiment of the present application provides a method for training a program information generation model, comprising:

[0006] Based on the first training data set, training an encoder of the program information generation model, wherein the encoder is used to perform feature extraction and alignment processing on the code and the comment in the input program information pair, respectively, to obtain a code vector corresponding to the code and a text vector corresponding to the comment;

[0007] Based on the encoder obtained through training, the program information pairs in the first training data set are encoded to obtain a text vector and a code vector corresponding to each program information pair;

[0008] Training a decoder of the program information generation model based on the text vector and the code vector corresponding to each pair of program information;

[0009] Based on the trained encoder and decoder, a program information generation model is constructed.

[0010] In a second aspect, an embodiment of the present application provides a program information generation model training device, comprising:

[0011] An encoder training module, used for training an encoder of the program information generation model based on a first training data set, wherein the encoder is used for performing feature extraction and alignment processing on the code and the comment in the input program information pair, respectively, to obtain a code vector corresponding to the code and a text vector corresponding to the comment;

[0012] An encoding processing module, configured to perform encoding processing on the program information pairs in the first training data set based on the encoder obtained through training, so as to obtain a text vector and a code vector corresponding to each program information pair;

[0013] A decoder training module, used for training a decoder of the program information generation model based on the text vector and the code vector corresponding to each pair of program information;

[0014] The model construction module is used to construct a program information generation model based on the encoder and the decoder obtained through training.

[0015] In a third aspect, an embodiment of the present application further discloses an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the program information generation model training method described in the embodiment of the present application is implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, which, when executed by a processor, implements the steps of the program information generation model training method disclosed in the embodiment of the present application.

[0017] The program information generation model training method disclosed in the embodiment of the present application is based on a first training data set, and the encoder of the program information generation model is trained, wherein the encoder is used to perform feature extraction and alignment processing on the code and the comment in the input program information pair, respectively, to obtain the code vector corresponding to the code and the text vector corresponding to the comment; based on the encoder obtained by training, the program information pairs in the first training data set are encoded to obtain the text vector and code vector corresponding to each program information pair; based on the text vector and the code vector corresponding to each program information pair, the decoder of the program information generation model is trained; based on the encoder and the decoder obtained by training, the program information generation model is constructed. Since the method uses feature alignment of comments and codes for feature extraction, the code and comment features obtained by the model during feature extraction can be mapped to the same space to a great extent, and the corresponding comments and code features have a high degree of similarity, that is, the code and the corresponding comments are output as similar vector matrices after passing through the program information generation model, which can better improve the semantic understanding of the model to the code, thereby improving the accuracy of the model generating code based on comments.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0020] Figure 1 It is one of the flow charts of the program information generation model training method disclosed in the embodiment of the present application;

[0021] Figure 2 It is a flow chart of the model training method disclosed in the embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the structure of the program information generation model disclosed in the embodiment of the present application;

[0023] Figure 4 It is a schematic diagram of the encoder training principle disclosed in the embodiment of the present application;

[0024] Figure 5 It is a schematic diagram of the decoder training principle disclosed in the embodiment of the present application;

[0025] Figure 6 This is the second flow chart of the program information generation model training method disclosed in the embodiment of the present application;

[0026] Figure 7 It is one of the structural schematic diagrams of the program information generation model training device disclosed in the embodiment of the present application;

[0027] Figure 8 This is the second structural diagram of the program information generation model training device disclosed in the embodiment of the present application;

[0028] Fig. 9 A block diagram schematically shows an electronic device for executing the method according to the present application; and

[0029] Fig.10 A storage unit for holding or carrying a program code for implementing the method according to the present application is schematically shown. DETAILED DESCRIPTION

[0030] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0031] In an embodiment of the present application, in order to solve the problem of low accuracy of generated program code, a training annotation and code alignment model (CCMM, Code Comment Matching Model) is used as an encoder, and a decoder for generating annotations and a decoder for generating code are trained to train a program information generation model. The program information generation model can generate code based on the input comments and generate comments based on the input code. Through the alignment training of code and comments, richer related information is introduced into the process of generating comments and code, thereby improving the accuracy of comment and code generation.

[0032] Furthermore, in order to improve the accuracy of annotation and code generation and standardize the encoder and decoder outputs, in an embodiment of the present application, reinforcement learning based on human feedback is used to perform proximal strategy optimization on the program information generation model.

[0033] The overall model training idea of ​​this application is as follows Figure 2 As shown. LM0 represents the basic language model, which can be trained based on the public data in the prior art so that the model has basic text understanding ability; LM represents the program information generation model; LM2 represents the optimized program information generation model, and RM represents the reward model. Figure 2 It can be seen that the program information generation model is the basic model. Figure 2 In , D1 represents a first training data set, and D2 represents a second training data set, wherein the second training data set is obtained based on the first training data set.

[0034] Next, the training method of the program information generation model is first explained.

[0035] Reference Figure 1 , a program information generation model training method disclosed in an embodiment of the present application includes: steps 110 to 140.

[0036] Step 110, based on the first training data set, train the encoder of the program information generation model, wherein the encoder is used to perform feature extraction and alignment processing on the code and comments in the input program information pair, respectively, to obtain the code vector corresponding to the code and the text vector corresponding to the comment.

[0037] Optionally, the first training data set is obtained by collecting questions and top answers from public data (e.g., from web pages, forums, knowledge bases, etc.), and after data cleaning and removing redundant information and interference information, obtaining the description text of the question and its corresponding code segment. Afterwards, the description text of the question is used as an annotation, and the code segment corresponding to the question is associated to generate a number of annotation and code pairs to constitute the first training data set. In the embodiment of the present application, the annotation and code pair is also referred to as a "program information pair".

[0038] On the other hand, for each comment and code pair in the first training data set, the code part can be segmented out using a named entity recognition method to identify special tokens (i.e., identifiers) in the code, such as code instructions such as def, class, return, etc. These special tokens are added to the dictionary and assigned special indexes. Then, the code part is vectorized based on the above dictionary so that the vectorized code can be input into the code feature extractor for feature extraction.

[0039] Optionally, for annotations, you can use the GPT dictionary for vectorization.

[0040] In an embodiment of the present application, the vectorized comments and codes may be separated by special marks (such as [SEP]), and each piece of data in the first training data set may be processed into a comment and code pair in the form of a text vector-code vector.

[0041] Reference Figure 3 , the program information generation model described in the embodiment of the present application includes: an encoder and a decoder. The encoder is used to extract the features of the comments and codes input to the program information generation model, and perform certain processing on the extracted features to obtain text vectors and code vectors; the decoder is used to decode the text vectors and code vectors output by the encoder. The decoded output includes: the code vector corresponding to the comment text input to the program information generation model, and the text vector corresponding to the code input to the program information generation model.

[0042] Reference Figure 4The encoder includes: a code feature extractor, an annotation feature extractor and a feature projection layer. The encoder for training the program information generation model based on the first training data set includes: extracting features from the code in the program information pair by the code feature extractor to obtain code features, and extracting features from the annotations in the corresponding program information pair by the annotation feature extractor to obtain annotation features; aligning and projecting the code features by the feature projection layer to obtain code vectors, and aligning and projecting the annotation features by the feature projection layer to obtain text vectors; obtaining the cosine similarity between the text vector and the code vector as the similarity corresponding to the program information pair; calculating the cross entropy loss based on the similarity as the encoding loss of the program information generation model; iteratively training the encoder with the goal of optimizing the encoding loss.

[0043] Optionally, the code feature extractor and the annotation feature extractor can respectively adopt the encoder structure in the Transfomer (transformer) model, and the feature projection layer can be a fully connected layer. The annotations and codes in the program information pair input to the encoder are respectively input to the corresponding feature extractors for feature extraction. For example, the code is input to the code feature extractor (referred to as "Tcode" in the embodiment of the present application) for feature extraction to obtain code features. On the other hand, the annotation is input to the annotation feature extractor (referred to as "Ttext" in the embodiment of the present application) for feature extraction to obtain annotation features. Then, the annotation features corresponding to the annotations and the code features corresponding to the code belonging to the same program information pair are input to the same feature projection layer, and the code features and annotation features are projected to the same vector space by the feature projection layer for alignment processing to obtain the code vector corresponding to the code features and the text vector corresponding to the annotation features, respectively.

[0044] Next, the encoding loss corresponding to the corresponding program information pair is calculated based on the text vector and the code vector output by the feature projection layer, thereby further calculating the encoding loss of the encoder based on the encoding losses corresponding to all program information pairs in the first training data set.

[0045] In an embodiment of the present application, the first training data set can be divided into multiple batches (multiple batches of training data), each batch has multiple program information pairs, and the codes and comments in each batch are combined into new samples in pairs, and the relevant codes and comments form positive samples, and the irrelevant codes and comments form negative samples. The higher the cosine similarity between the code vector and the text vector corresponding to the positive sample, the better (close to 1), and the lower the cosine similarity between the code vector and the text vector corresponding to the negative sample, the better (close to 0). For the newly composed batch samples, when training the encoder, each batch data can be annotated as a two-dimensional array, the identification bit corresponding to the positive sample is 1, the identification bit corresponding to the negative sample is 0, and the array dimension value is the number of samples in the batch. In the prior art, since the text part already has good Transformer feature extraction parameters, when training the encoder, in order to reduce the amount of updated parameters, the parameters of the annotation feature extractor can not be updated, or, after a certain round of training, some parameters of the annotation feature extractor and the code feature extractor are not updated, and only the parameters of the feature projection layer with fewer parameters are updated to accelerate the model training of the program information generation model.

[0046] In some embodiments of the present application, the cross entropy loss is calculated by calculating the processed code and the annotations to obtain the encoding loss of the encoder. For example, the encoding loss can be calculated by the following formula:

[0047]

[0048] Among them, θ represents the model parameters of the encoder, c and w are the code features and comment features after L2 normalization (Euclidean normalization), D c,θ (T text ) represents a text vector obtained after the feature projection layer performs feature projection on the annotation features extracted by the annotation feature extractor; D w,θ (T code ) represents the code vector obtained after the feature projection layer performs feature projection on the code features extracted by the code feature extractor, Sim() is the cosine similarity calculation function, i represents the axis, i=0 represents the row axis, i=1 represents the column axis, j represents the index of the program information pair, and They represent the cross entropy loss on different axes, namely the encoding loss of text features and the encoding loss of code features.

[0049] Furthermore, with the goal of optimizing the coding loss, the parameters of the decoder are adjusted, and the encoder is iteratively trained until the coding loss converges or a preset training end condition is reached, thereby completing the encoder training.

[0050] In other embodiments of the present application, other formulas may also be used to calculate the cross entropy loss based on the cosine similarity of the text vector and the code vector to obtain the encoding loss of the program information generation model, which are not listed one by one in the embodiments of the present application.

[0051] Step 120: encode the program information pairs in the first training data set based on the encoder obtained through training to obtain a text vector and a code vector corresponding to each program information pair.

[0052] After the encoder training is completed, the encoder parameters can be frozen and only the decoder parameters can be fine-tuned to train the decoder.

[0053] In the process of training the decoder, the program information pairs in the second training data set are first input into the encoder, and the encoder encodes the codes and comments in the input program information pairs respectively to obtain a set of similar text vectors and code vectors corresponding to the program information pairs.

[0054] Step 130: training a decoder of the program information generation model based on the text vector and the code vector corresponding to each pair of the program information.

[0055] Next, the text vector and code vector corresponding to each program information pair are concatenated and used as the input of the decoder.

[0056] The decoder is used to perform spatial projection on the text vector and code vector output by the encoder, and decode the corresponding code vector and text vector respectively through the first decoder and the second decoder built into the decoder to obtain a decoded output.

[0057] Reference Figure 5 The decoder includes: a first decoder and a second decoder, the first decoder is used to decode the output code prediction result, and the second decoder is used to decode the output comment prediction result.

[0058] In some optional embodiments, the decoder of the program information generation model is trained based on the text vector and the code vector corresponding to each program information pair, including: performing multiple rounds of progressive decoding mapping on the annotation-code splicing vector corresponding to the program information pair by the first decoder to obtain the code prediction result corresponding to the program information pair, wherein the annotation-code splicing vector is obtained by splicing part of the code vector with the text vector; and performing decoding mapping on the code vector corresponding to each program information pair by the second decoder to obtain the annotation prediction result corresponding to the program information pair; obtaining the annotation decoding loss and the code decoding loss respectively based on the code prediction result and the annotation prediction result; obtaining the decoding loss of the decoder based on the annotation decoding loss and the code decoding loss; and iteratively training the decoder with the goal of optimizing the decoding loss.

[0059] In some optional embodiments, the first decoder performs multiple rounds of progressive decoding mapping on the annotation code splicing vector corresponding to the program information pair to obtain the code prediction result corresponding to the program information pair, including: splicing the text vector with the first part of the code vector to obtain the annotation code splicing vector, wherein the first part of the code vector is a vector of a preset length at the head of the code vector; using the annotation code splicing vector and the code prediction result obtained by the previous round of decoding mapping as the current round decoding input vector of the first decoder, and using each dimensional vector in the second part of the code vector as the target of the current round decoding prediction, and decoding and mapping each round of decoding input vector in turn by the first decoder to obtain the corresponding round of code prediction result; and obtaining the corresponding code prediction result of the program information pair according to each round of code prediction result.

[0060] Optionally, during the decoder training process, the following processing is performed on both the input vector and the label to perform decoding prediction.

[0061] For the first decoder, the complete text vector can be input. At the same time, only the first part of the code vector is used as the input of the first decoder together with the text vector. Each token (i.e., vector value) in the second part of the code vector is used as the target of the first decoder prediction. The output and input of this time are simultaneously input to the first decoder to obtain the probability distribution prediction value of the next token until the next token is predicted to be the last one (e.g., the next token is the END mark), which means that the decoding process of the current text vector is stopped. For example, the complete text vector is represented as w = {w,...,w T}, the total code vector is represented as c = {c,...,c T For example, one round of decoding input can be expressed as: i,w={w,...,w T ,c 1 ,...,c K},k<=T.

[0062] For the second decoder, the complete code vector can be input without inputting the text vector.

[0063] Correspondingly, the decoding loss of the decoder can be calculated by the following formula:

[0064]

[0065] Where θ represents the decoder parameters, D represents the first training data set, represents the decoding loss of the first decoder, represents the decoding loss of the second decoder, α i represents the decoding loss weights of the first decoder and the second decoder, T represents the number of dimensions of the text vector, P θ (w|c) represents the probability that the decoding prediction result is annotation W under the condition that the input is code vector c, P θ (c T-I |c i ,w) indicates that the input is a complete text vector and a partial code vector c i The probability that the decoding prediction result is the next-dimensional code vector under the condition .

[0066] Next, with the goal of optimizing the decoding loss, the decoder is iteratively trained until the decoding loss converges, thereby completing the decoder training.

[0067] In the embodiments of the present application, other formulas may also be used to calculate the overall loss of the decoder according to the decoding loss of the first decoder and the decoding loss of the second decoder, which will not be listed one by one in the embodiments of the present application.

[0068] Step 140: construct a program information generation model based on the trained encoder and decoder.

[0069] After the encoder and decoder are trained, the program information generation model is trained.

[0070] In some optional embodiments, such as Figure 6 As shown, after constructing a program information generation model based on the encoder and the decoder obtained through training, the method further includes: step 150 and step 160.

[0071] Step 150, training the reward model.

[0072] The training reward model further includes: sub-step A1, sub-step A2 and sub-step A3.

[0073] Sub-step A1, calling the program information generation model to generate a second training data set.

[0074] The second training data set includes: an annotation sequence corresponding to the code, and a code sequence corresponding to the annotation, wherein the annotation sequence includes: K annotations sorted by quality scores, and the code sequence includes: K segments of code sorted by quality scores.

[0075] In an embodiment of the present application, the second training data set for training the reward model mainly consists of a generated data set and a manually annotated data set. Among them, the data of the manually annotated data set comes from the generated data set. The generated data set contains codes, comments and corresponding answers. Among them, the corresponding answer can be the comment corresponding to the code, or the code corresponding to the comment. The comment is a complete sentence, and the code and the comment have a certain correlation.

[0076] In order to obtain the answer, the code and comments in the first training data set must first be input into the program information generation model trained in the above steps for processing. In an embodiment of the application, a beam search method can be used to generate K different answers corresponding to the same comment, that is, the data format of the generated data set is: one comment corresponds to K code segments, and one code segment corresponds to K comments. The manually annotated data set is derived from the generated data set. The specific method is: manually sort the answer part of the generated data set according to its quality, that is, sort the multiple answers corresponding to one comment. In this way, the complexity of the manually annotated data set can be reduced.

[0077] In an embodiment of the present application, the reward model is obtained by modifying the program information generation model. The reward model includes: an encoder trained by the aforementioned steps, and on the other hand, the output of the decoder of the program information generation model is modified to output a quality score. For example, the output conversion layer of the decoder for outputting a vector representation of a code or comment statement is replaced with an initialized linear layer, and the linear layer is used to output a quality score. The quality score represents the quality of the answer, and a higher score represents a higher encoding and decoding quality.

[0078] Next, a reward model may be trained by performing a variety of tasks based on the second training data set.

[0079] Sub-step A2: Based on the second training data set, at least the following first training task and second training task are performed to train the reward model.

[0080] Among them, the first training task is used to process the annotations in the second training data set through the reward model, obtain the reward value corresponding to the first code and the reward value corresponding to the second code, and iteratively optimize the reward model with the goal of maximizing the difference between the reward value corresponding to the first code and the reward value corresponding to the second code, wherein the first code includes: the first W segments of code corresponding to the corresponding annotations with quality scores from high to low, and the second code includes: the first KW segments of code corresponding to the corresponding annotations with quality scores from low to high.

[0081] For example, the comment and the corresponding answer in the second data set are input into the reward model. At this time, the answer is the code corresponding to the comment. Since the manually annotated data set is a comment corresponding to K code segments as answers, for a piece of data in the second training data set, it needs to be repeated K times to obtain the quality scores of K answers. The purpose of model training is to make the K quality scores meet the ranking of the manually annotated data set.

[0082] Taking the input of the reward model as the annotation x in the second training dataset as an example, the K answers corresponding to the annotation x in the second training dataset can be expressed as: y = {y 1 ,...,y K}, where the quality score ranking order is y 1 >y 2 >…>y K , the high quality score code is taken as the “first code” and represented as y w ,y w ={y 1 ,...,y w}, the code with lower quality score is regarded as "second code", and the answer corresponding to the comment is represented as y 1 ,y 1 ={y w+1 ,...,y K}. K represents the number of answers corresponding to each comment. θ (x,y w ) represents the reward value obtained by inputting the annotation and high-quality answer into the reward model, expressed as r θ (x,y l ) represents the reward value obtained by inputting the annotation and the answer with lower quality score into the reward model. The purpose of the first training task is to make the difference between the high-quality and low-quality answers as large as possible. Based on this, the loss value of the first training task can be calculated by the following formula:

[0083]

[0084] in, Refers to selecting annotations x and answers y from the manually labeled dataset Dw ,y l Calculate the expected value after cross entropy, σ() function is sigmoid function; represents the number of reward value comparisons, and D represents the second training data set.

[0085] Among them, the second training task is used to process the program information pairs in the second training data set through the reward model, obtain the reward value of the first program information pair and the reward value of the second program information pair, and iteratively optimize the reward model with the goal of maximizing the difference between the reward value of the first program information pair and the reward value of the second program information pair, wherein the first program information pair includes: each code and the annotated code pair consisting of the top W annotations corresponding to the code with quality scores from high to low, and the second program information pair includes: each code and the annotated code pair consisting of the top KW annotations corresponding to the code with quality scores from low to high; wherein K is an integer greater than 1, and W is a positive integer less than K.

[0086] For example, the comments, codes and corresponding answers in the second data set are input into the reward model. Since the manually annotated data set is a comment corresponding to K code segments as answers, for one of the above data in the second training data set, it is necessary to repeat K times to obtain the quality scores of K answers. The purpose of model training is to make the K quality scores meet the ranking of the manually annotated data set.

[0087] Taking the input of the reward model as the program information pair (x, c) in the second training data set as an example, where x represents the comment and c represents the code corresponding to the comment, the K answers corresponding to the program information pair (x, c) in the second training data set can be expressed as: y = {y 1 ,...,y K}, where the quality score ranking order is y 1 >y 2 >…>y K , the program information pair with high quality score is regarded as the "first program information pair", and its corresponding answer is represented as y w ,y w ={y 1 ,...,y w}, the program information pair with a lower quality score is regarded as the “second program information pair”, and its corresponding answer is represented as y 1 ,y 1 ={y w+1 ,...,y K}. θ (x,c,y w ) represents the reward value obtained by inputting the program information pair and the high-quality answer into the reward model, with r θ(x,c,y l ) represents the reward value obtained by inputting the program information pair and the answer with a lower quality score into the reward model. The purpose of the second training task is to make the difference between the high-quality and low-quality answers as large as possible. Based on this, the loss value of the second training task can be calculated by the following formula:

[0088]

[0089] in, Refers to selecting annotation x, code c, and answer y from the manually labeled dataset D x ,y l Calculate the expected value after cross entropy, σ() function is sigmoid function.

[0090] After training the reward model by executing the above-mentioned first training task and / or second training task, the trained reward model can identify the quality of the answers generated by the program information generation model, that is, based on the input code and comments, the reward model can judge the quality of the code segments or comments generated by the program information generation model.

[0091] Step 160: Optimize the program information generation model based on the reward model.

[0092] Optionally, the program information generation model is optimized based on the reward model.

[0093] Based on the second training data set, at least the following first training task and second training task are performed to train the reward model and then performed. The step of optimizing the program information generation model based on the reward model includes: sub-step B1, sub-step B2 and sub-step B3.

[0094] Sub-step B1, obtaining a third training data set, wherein the third training data set includes: program information pairs.

[0095] Optionally, the third training data set is obtained by expanding the second training data set. For example, the annotation and code pairs in the second training data set generated when training the reward model can be used as part of the third training data set. On the other hand, a large amount of corpus can be selected from other corpora to add to the data set, including but not limited to codes, annotation statements corresponding to the codes, etc. Then, the codes and annotation statements are processed according to the data processing method described above to generate new annotation and code pairs to expand the program information pairs in the third training data set.

[0096] Sub-step B2, generating an answer corresponding to the program information through the program information generation model to be optimized.

[0097] Next, the program information pair (x, c) in the third training data set is input into the program information generation model, and the answer y corresponding to the program information pair (x, c) generated by the program information generation model is obtained.

[0098] Sub-step B3, inputting the program information pair and the generated answer into the reward model, obtaining a reward score, and fine-tuning the program information generation model to be optimized based on the reward score.

[0099] For example, the program information pair (x, c) and its corresponding answer y are input into the reward model together. After the reward model processes the input (x, c, y), it can output a reward score r corresponding to (x, c, y). θ . Further, based on the reward score r θ The program information generation model to be optimized is fine-tuned. Optionally, the program information generation model to be optimized is restricted using KL divergence so that it cannot deviate greatly from the original program information generation model in order to obtain a higher score.

[0100] The formula for fine-tuning the program information generation model to be optimized based on the reward score can be expressed as follows:

[0101]

[0102] Wherein, x, c, y represent the associated comments and codes input to the reward model and the output of the program information generation model to be optimized for the comments and codes respectively; r θ (x,c,y) represents the reward score output by the reward model; represents the program information generation model to be optimized; LM represents the program information generation model; β, γ are adjustment coefficients; is a set of the third training data set and the corresponding outputs generated by the third training data set; D pretrain Represents the second training data set.

[0103] Since code generation requires high precision, the model is optimized by using reinforcement learning based on human feedback and proximal policy optimization algorithms in the embodiments of the present application, which helps to standardize the output of the model. Furthermore, the model is trained by using training tasks that are more suitable for the model application scenario, and the training tasks are more suitable for the actual application scenarios of code generation, so that the model can also achieve better performance in small sample tasks.

[0104] In some optional embodiments, after constructing a program information generation model based on the encoder and the decoder obtained through training, the method further includes: encoding the program information to be processed by the encoder of the program information generation model to obtain a text vector and / or a code vector of the program information to be processed; decoding the text vector and / or the code vector by the decoder of the program information generation model to obtain a code prediction result corresponding to the text vector and / or an annotation prediction result corresponding to the code vector.

[0105] Among them, the encoder of the program information generation model encodes the program information to be processed to obtain a text vector and / or a code vector of the program information to be processed, including: performing feature extraction on the code in the program information to be processed by the code feature extractor in the encoder of the program information generation model to obtain code features, and / or, performing feature extraction on the comments in the program information to be processed by the comment feature extractor in the encoder to obtain comment features; performing feature alignment projection on the code features by the feature projection layer to obtain a code vector, and / or, performing feature alignment projection on the comment features by the feature projection layer to obtain a text vector.

[0106] Among them, the text vector and / or the code vector are decoded by the decoder of the program information generation model to obtain the code prediction result corresponding to the text vector and / or the annotation prediction result corresponding to the code vector, including: performing multiple rounds of progressive decoding and mapping on the annotation-code splicing vector of the annotation by the first decoder to obtain the code prediction result corresponding to the program information to be processed, wherein the annotation-code splicing vector is obtained by splicing part of the code vector with the text vector; and / or, decoding and mapping the code vector by the second decoder to obtain the annotation prediction result of the program information to be processed.

[0107] The specific method for the encoder to perform encoding processing is referred to the relevant description in the above embodiment, which will not be repeated here. The specific method for the decoder to perform decoding processing is referred to the relevant description in the above embodiment, which will not be repeated here.

[0108] The program information generation model trained by the program information generation model training method disclosed in the embodiments of the present application can realize code generation and annotation generation in one model, and the model utilization rate is higher.

[0109] In summary, the program information generation model training method disclosed in the embodiment of the present application is to train the encoder of the program information generation model based on the first training data set, wherein the encoder is used to perform feature extraction and alignment processing on the code and the comment in the input program information pair respectively, and obtain the code vector corresponding to the code and the text vector corresponding to the comment; based on the encoder obtained by training, the program information pairs in the first training data set are encoded to obtain the text vector and code vector corresponding to each program information pair; based on the text vector and the code vector corresponding to each program information pair, the decoder of the program information generation model is trained; based on the encoder and the decoder obtained by training, the program information generation model is constructed. This method adopts the feature alignment method of comments and codes, and uses it to train the feature extraction part of the model, so that the code and comment features obtained by the model during feature extraction can be mapped to the same space to a great extent, and the corresponding comments and code features have a high similarity, that is, the code and the corresponding comments are output as similar vector matrices after passing through the program information generation model, which can better improve the semantic understanding of the model to the code, thereby improving the accuracy of the model generating code based on the comments.

[0110] At the same time, highly similar code and comment features can make the text richer in information for clustering, so the program information generation model can achieve good performance in the tasks of outputting comment text based on code and outputting corresponding code based on comment text.

[0111] Furthermore, since the model is trained with training tasks that fit the application scenario, the trained program information generation model has good performance for specific code generation scenarios and comment generation scenarios. Compared with the existing code generation system, each input comment is given a clearer feature and contains more information. By adding code features, code feature information is added to each comment word to make the description more precise. On the other hand, since the code features contain comment information, the generated code will be more in line with the input requirements.

[0112] Based on the above embodiments, the present application also discloses a program information generation model training device, see Figure 7 , the device comprises:

[0113] An encoder training module 710 is used to train an encoder of the program information generation model based on a first training data set, wherein the encoder is used to perform feature extraction and alignment processing on the code and the comment in the input program information pair, respectively, to obtain a code vector corresponding to the code and a text vector corresponding to the comment;

[0114] An encoding processing module 720, configured to perform encoding processing on the program information pairs in the first training data set based on the encoder obtained through training, to obtain a text vector and a code vector corresponding to each program information pair;

[0115] A decoder training module 730, configured to train a decoder of the program information generation model based on the text vector and the code vector corresponding to each pair of program information;

[0116] The model construction module 740 is used to construct a program information generation model based on the encoder and the decoder obtained through training.

[0117] Optionally, the encoder includes: a code feature extractor, an annotation feature extractor and a feature projection layer, and the encoder training module 710 is further used to:

[0118] Extracting features of the codes in the program information pair by the code feature extractor to obtain code features, and extracting features of the comments in the corresponding program information pair by the comment feature extractor to obtain comment features;

[0119] Performing feature alignment projection on the code feature through the feature projection layer to obtain a code vector, and performing feature alignment projection on the annotation feature through the feature projection layer to obtain a text vector;

[0120] Obtaining the cosine similarity between the text vector and the code vector as the similarity corresponding to the program information pair;

[0121] Calculate a cross entropy loss based on the similarity as the encoding loss of the program information generation model;

[0122] The encoder is iteratively trained with the goal of optimizing the encoding loss.

[0123] Optionally, the decoder includes: a first decoder and a second decoder, and the decoder training module 730 is further used to:

[0124] Performing multiple rounds of progressive decoding mapping on the annotation code splicing vector corresponding to the program information pair by the first decoder to obtain a code prediction result corresponding to the program information pair, wherein the annotation code splicing vector is obtained by splicing part of the code vector with the text vector; and

[0125] Decoding and mapping the code vectors corresponding to each of the program information pairs by the second decoder to obtain annotation prediction results corresponding to the program information pairs;

[0126] Based on the code prediction result and the annotation prediction result, respectively obtaining the annotation decoding loss and the code decoding loss;

[0127] Obtaining a decoding loss of the decoder based on the annotation decoding loss and the code decoding loss;

[0128] The decoder is iteratively trained with the goal of optimizing the decoding loss.

[0129] Optionally, performing multiple rounds of progressive decoding mapping on the annotation code splicing vector corresponding to the program information pair by the first decoder to obtain a code prediction result corresponding to the program information pair includes:

[0130] Splicing the text vector with the code vector in the first part to obtain an annotation code splicing vector, wherein the code vector in the first part is a vector of a preset length at the head of the code vector;

[0131] The code prediction result obtained by the annotation code concatenation vector and the previous round decoding mapping is used as the current round decoding input vector of the first decoder, and each dimensional vector in the code vector in the second part is used as the target of the current round decoding prediction. The first decoder decodes and maps the decoding input vectors of each round in turn to obtain the corresponding round code prediction result;

[0132] The code prediction result corresponding to the program information is obtained according to the code prediction results of each round.

[0133] Optional, see Figure 8 , the device further comprises:

[0134] The reward model training module 750 is used to call the program information generation model to generate a second training data set, wherein the second training data set includes: a comment sequence corresponding to the code, and a code sequence corresponding to the comment, wherein the comment sequence includes: K comments sorted by quality score, and the code sequence includes: K segments of code sorted by quality score; and, based on the second training data set, at least the following first training task and second training task are performed to train the reward model, wherein:

[0135] The first training task is used to process the annotations in the second training data set through the reward model, obtain the reward value corresponding to the first code and the reward value corresponding to the second code, and iteratively optimize the reward model with the goal of maximizing the difference between the reward value corresponding to the first code and the reward value corresponding to the second code, wherein the first code includes: the first W segments of code corresponding to the corresponding annotations with quality scores from high to low, and the second code includes: the first KW segments of code corresponding to the corresponding annotations with quality scores from low to high;

[0136] The second training task is used to process the program information pairs in the second training data set through the reward model, obtain the reward value of the first program information pair and the reward value of the second program information pair, and iteratively optimize the reward model with the goal of maximizing the difference between the reward value of the first program information pair and the reward value of the second program information pair, wherein the first program information pair includes: each code and an annotated code pair consisting of the top W annotations corresponding to the code with quality scores from high to low, and the second program information pair includes: each code and an annotated code pair consisting of the top KW annotations corresponding to the code with quality scores from low to high;

[0137] Wherein, K is an integer greater than 1, and W is a positive integer less than K.

[0138] Optional, see Figure 8 , the device further comprises:

[0139] The model optimization module 760 is used to obtain a third training data set, wherein the third training data set includes: program information pairs;

[0140] The model optimization module 760 is also used to generate the answer corresponding to the program information pair through the program information generation model to be optimized; and input the program information pair and the generated answer into the reward model to obtain a reward score; and then fine-tune the program information generation model to be optimized based on the reward score.

[0141] Optionally, the device further comprises:

[0142] a program information generation module, configured to encode the program information to be processed by the encoder of the program information generation model to obtain a text vector and / or a code vector of the program information to be processed; and

[0143] The text vector and / or the code vector are decoded by the decoder of the program information generation model to obtain a code prediction result corresponding to the text vector and / or an annotation prediction result corresponding to the code vector.

[0144] The program information generation model training device disclosed in the embodiment of the present application is used to implement the program information generation model training method described in the embodiment of the present application. The specific implementation methods of each module of the device will not be repeated here, and reference can be made to the specific implementation methods of the corresponding steps in the method embodiment.

[0145] The program information generation model training device disclosed in the embodiment of the present application is based on a first training data set to train the encoder of the program information generation model, wherein the encoder is used to perform feature extraction and alignment processing on the code and the comment in the input program information pair, respectively, to obtain the code vector corresponding to the code and the text vector corresponding to the comment; based on the encoder obtained by training, the program information pairs in the first training data set are encoded to obtain the text vector and code vector corresponding to each program information pair; based on the text vector and the code vector corresponding to each program information pair, the decoder of the program information generation model is trained; based on the encoder and the decoder obtained by training, the program information generation model is constructed. This method adopts the feature alignment method of comments and codes, and uses it to train the feature extraction part of the model, so that the code and comment features obtained by the model during feature extraction can be mapped to the same space to a great extent, and the corresponding comment and code features have a high similarity, that is, the code and the corresponding comment are output as similar vector matrices after passing through the program information generation model, which can better improve the semantic understanding of the model to the code, thereby improving the accuracy of the model generating code based on the comments.

[0146] At the same time, highly similar code and comment features can make the text richer in information for clustering, so the program information generation model can achieve good performance in the tasks of outputting comment text based on code and outputting corresponding code based on comment text.

[0147] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0148] The above is a detailed introduction to a program information generation model training method and device provided by the present application. This article uses specific examples to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0149] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0150] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the electronic device according to the embodiment of the present application. The present application can also be implemented as a device or apparatus program (e.g., computer program and computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0151] For example, Fig. 9 An electronic device that can implement the method according to the present application is shown. The electronic device can be a PC, a mobile terminal, a personal digital assistant, a tablet computer, etc. The electronic device traditionally includes a processor 910 and a memory 920 and a program code 930 stored on the memory 920 and can be run on the processor 910, and the processor 910 implements the method described in the above embodiment when executing the program code 930. The memory 920 can be a computer program product or a computer-readable medium. The memory 920 can be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk or a ROM. The memory 920 has a storage space 9201 for the program code 930 of the computer program for executing any method step in the above method. For example, the storage space 9201 for the program code 930 can include various computer programs for implementing the various steps in the above method respectively. The program code 930 is a computer-readable code. These computer programs can be read from one or more computer program products or written into the one or more computer program products. These computer program products include program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. The computer program includes a computer readable code, and when the computer readable code is executed on an electronic device, the electronic device is caused to execute the method according to the above embodiment.

[0152] An embodiment of the present application also discloses a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the program information generation model training method described in the embodiment of the present application are implemented.

[0153] Such a computer program product may be a computer-readable storage medium, which may have a computer-readable storage medium having a Fig. 9 The memory 920 in the electronic device shown in the figure is similar to the storage segment, storage space, etc. The program code can be compressed and stored in the computer-readable storage medium in an appropriate form. The computer-readable storage medium is generally as shown in the reference Fig.10 The portable or fixed storage unit. Generally, the storage unit includes computer readable code 930', which is a code read by a processor. When the code is executed by the processor, each step in the method described above is implemented.

[0154] References to "one embodiment," "embodiment," or "one or more embodiments" herein mean that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least one embodiment of the present application. In addition, please note that examples of the term "in one embodiment" herein do not necessarily all refer to the same embodiment.

[0155] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0156] In the claims, any reference signs placed between brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application may be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0157] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for training a program information generation model, characterized in that: The method comprises: Based on the first training data set, training an encoder of the program information generation model, wherein the encoder is used to perform feature extraction and alignment processing on the code and the comment in the input program information pair, respectively, to obtain a code vector corresponding to the code and a text vector corresponding to the comment; Based on the encoder obtained through training, the program information pairs in the first training data set are encoded to obtain a text vector and a code vector corresponding to each program information pair; Training a decoder of the program information generation model based on the text vector and the code vector corresponding to each pair of program information; Based on the trained encoder and decoder, a program information generation model is constructed.

2. The method according to claim 1, characterized in that The encoder includes: a code feature extractor, an annotation feature extractor and a feature projection layer, and the encoder for training the program information generation model based on the first training data set includes: Extracting features of the codes in the program information pair by the code feature extractor to obtain code features, and extracting features of the comments in the corresponding program information pair by the comment feature extractor to obtain comment features; Performing feature alignment projection on the code feature through the feature projection layer to obtain a code vector, and performing feature alignment projection on the annotation feature through the feature projection layer to obtain a text vector; Obtaining the cosine similarity between the text vector and the code vector as the similarity corresponding to the program information pair; Calculate a cross entropy loss based on the similarity as the encoding loss of the program information generation model; The encoder is iteratively trained with the goal of optimizing the encoding loss.

3. The method according to claim 1, characterized in that: The decoder comprises: a first decoder and a second decoder, wherein the decoder for training the program information generation model based on the text vector and the code vector corresponding to each pair of program information comprises: Performing multiple rounds of progressive decoding mapping on the annotation code splicing vector corresponding to the program information pair by the first decoder to obtain a code prediction result corresponding to the program information pair, wherein the annotation code splicing vector is obtained by splicing part of the code vector with the text vector; and Decoding and mapping the code vectors corresponding to each of the program information pairs by the second decoder to obtain annotation prediction results corresponding to the program information pairs; Based on the code prediction result and the annotation prediction result, respectively obtaining the annotation decoding loss and the code decoding loss; Obtaining a decoding loss of the decoder based on the annotation decoding loss and the code decoding loss; The decoder is iteratively trained with the goal of optimizing the decoding loss.

4. The method according to claim 3, characterized in that The step of performing multiple rounds of progressive decoding mapping on the annotation code splicing vector corresponding to the program information pair by the first decoder to obtain a code prediction result corresponding to the program information pair includes: Splicing the text vector with the code vector in the first part to obtain an annotation code splicing vector, wherein the code vector in the first part is a vector of a preset length at the head of the code vector; The code prediction result obtained by the annotation code concatenation vector and the previous round decoding mapping is used as the current round decoding input vector of the first decoder, and each dimensional vector in the code vector in the second part is used as the target of the current round decoding prediction. The first decoder decodes and maps the decoding input vectors of each round in turn to obtain the corresponding round code prediction result; The code prediction result corresponding to the program information is obtained according to the code prediction results of each round.

5. The method according to claim 1, characterized in that After constructing the program information generation model based on the encoder and the decoder obtained through training, the method further includes: Calling the program information generation model to generate a second training data set, the second training data set includes: an annotation sequence corresponding to the code, and a code sequence corresponding to the annotation, the annotation sequence includes: K annotations sorted by quality score, and the code sequence includes: K segments of code sorted by quality score; Based on the second training data set, at least the following first training task and second training task are performed to train the reward model, wherein: The first training task is used to process the annotations in the second training data set through the reward model, obtain the reward value corresponding to the first code and the reward value corresponding to the second code, and iteratively optimize the reward model with the goal of maximizing the difference between the reward value corresponding to the first code and the reward value corresponding to the second code, wherein the first code includes: the first W segments of code corresponding to the corresponding annotations with quality scores from high to low, and the second code includes: the first KW segments of code corresponding to the corresponding annotations with quality scores from low to high; The second training task is used to process the program information pairs in the second training data set through the reward model, obtain the reward value of the first program information pair and the reward value of the second program information pair, and iteratively optimize the reward model with the goal of maximizing the difference between the reward value of the first program information pair and the reward value of the second program information pair, wherein the first program information pair includes: each code and an annotated code pair consisting of the top W annotations corresponding to the code with quality scores from high to low, and the second program information pair includes: each code and an annotated code pair consisting of the top KW annotations corresponding to the code with quality scores from low to high; Wherein, K is an integer greater than 1, and W is a positive integer less than K.

6. The method according to claim 5, characterized in that After performing at least the following first training task and second training task based on the second training data set to train the reward model, the method further includes: Acquire a third training data set, wherein the third training data set includes: program information pairs; Generate an answer corresponding to the program information by using the program information generation model to be optimized; Inputting the program information pair and the generated answer into the reward model to obtain a reward score; The program information generation model to be optimized is fine-tuned based on the reward score.

7. The method according to claim 1, characterized in that After constructing the program information generation model based on the encoder and the decoder obtained through training, the method further includes: Encoding the program information to be processed by the encoder of the program information generation model to obtain a text vector and / or a code vector of the program information to be processed; The text vector and / or the code vector are decoded by the decoder of the program information generation model to obtain a code prediction result corresponding to the text vector and / or an annotation prediction result corresponding to the code vector.

8. A program information generation model training device, characterized in that: The device comprises: An encoder training module, used for training an encoder of the program information generation model based on a first training data set, wherein the encoder is used for performing feature extraction and alignment processing on the code and the comment in the input program information pair, respectively, to obtain a code vector corresponding to the code and a text vector corresponding to the comment; An encoding processing module, configured to perform encoding processing on the program information pairs in the first training data set based on the encoder obtained through training, so as to obtain a text vector and a code vector corresponding to each program information pair; A decoder training module, used for training a decoder of the program information generation model based on the text vector and the code vector corresponding to each pair of program information; A model construction module is used to construct a program information generation model based on the encoder and the decoder obtained through training.

9. An electronic device comprising a memory, a processor, and a program code stored in the memory and executable on the processor, wherein: When the processor executes the program code, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having program code stored thereon, characterized in that: When the program code is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

Images