A code pre-training method based on an adapter network and contrastive learning

Abstract

The application discloses a code pre-training method based on an Adapter network and contrast learning, and the method comprises the following steps: constructing a code pre-training model based on an Adapter network, obtaining training instances x in code corpus d , training the code pre-training model based on the Adapter network to obtain an Adapter network with different programming language knowledge; constructing a low-resource programming language model; obtaining instances x in the code corpus c , respectively constructing corresponding positive examples and negative examples, jointly training the low-resource programming language model by using classification cost and contrast learning cost; training a final code pre-training model, and outputting a code pre-training result based on the final code pre-training model. The application can reduce the problem of catastrophic forgetting of existing knowledge while pre-training code corpus of multiple programming languages, support training of new programming languages, and enhance the model capability under low-resource programming languages.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of code pre-training, and particularly relates to a code pre-training method based on an Adapter network and contrastive learning. BACKGROUND

[0002] The purpose of code generation task is to automatically generate executable code according to the input high-level description (usually a paragraph of natural language, describing the intention of human). The automatically generated code can be a complete application program, a system component, or only some specific modules, functions, or code fragments. Automatic code generation can reduce repetitive programming tasks and help developers focus on more innovative and complex business logic. By automatically generating common code templates or fragments, the time of manual writing is reduced, thereby greatly improving the development efficiency of programmers.

[0003] The automatic code generation method can be roughly divided into the following two categories: 1) sequence-to-sequence network based method: based on the encoder-decoder network architecture and the attention mechanism, the corresponding relationship between the natural language intention and the code is learned from the training data set. Specifically, the encoder is responsible for encoding the natural language intention into a fixed-length vector representation, which is the abstract representation information of the natural language intention, and the decoder decodes the executable code step by step according to the vector representation of the encoder. 2) large pre-training model based method: this method is divided into two stages, the pre-training stage is based on a large amount of code corpus of different programming languages to train a code pre-training model with general code knowledge, and the fine-tuning stage is based on the model obtained in the last stage to fine-tune the automatic code generation task.

[0004] At present, the sequence-to-sequence network based method is only trained based on a small amount of code generation task data, although it can learn part of the mapping relationship, but it seriously lacks the general understanding of the code itself, resulting in the performance generally lower than the large pre-training model based method, and the lack of artificially annotated training data about code generation often leads to the problem of overfitting of the sequence-to-sequence network based model.

[0005] And the following two shortcomings are based on large pre-training model: first, the method based on large pre-training model mixes the corpus of different code languages into the model for training, and then fine-tunes the pre-trained model for automatic code generation task. Therefore, after training the code pre-training model, if the model wants to increase the support for new code language, the whole model needs to be retrained by using new code language; and at the same time, integrating different languages will cause the problem of catastrophic forgetting, and the parameter quantity of the pre-training model is generally very large, so the retraining cost is high. Second, for some low-resource programming languages, the code corpus that can be obtained is very small, so compared with the high-resource pre-training model trained by a large amount of data, the model ability of low-resource programming language is weak.

[0006] When the existing method pre-trains the Transformer model in the mixed corpus of different programming languages, due to the fact that the Transformer model generally contains a large number of parameters and the number of code corpus of different programming languages is very different, for example, the proportion of Ruby programming language code in the whole corpus is very small, the existing method often forgets the knowledge of the programming language learned before, thereby the problem of catastrophic forgetting of existing knowledge occurs, and when a new programming language needs to be supported, the whole model needs to be retrained, which has a large training resource and time cost. SUMMARY

[0007] In order to overcome the defects and deficiencies existing in the prior art, the present application provides a code pre-training method based on Adapter network and contrastive learning, which constructs an Adapter network-based code pre-training model, trains an Adapter network with different programming language knowledge based on a large amount of different programming language corpus, can reduce the problem of catastrophic forgetting of existing knowledge while pre-training the code corpus of multiple programming languages, and support the training of new programming languages; construct positive and negative examples for low-resource programming language data, jointly train low-resource programming language model using classification cost and contrastive learning cost, which can enhance the model capability of low-resource programming language.

[0008] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0009] The present application provides a code pre-training method based on Adapter network and contrastive learning, comprising the following steps:

[0010] The code pre-training model based on the Adapter network is constructed, including a plurality of Transformer layers, Adapter networks corresponding to different programming languages, a fusion layer and a classification layer, the plurality of Transformer layers learn general knowledge of the programming languages, the Adapter networks learn domain knowledge of the current programming language, the fusion layer fuses the general knowledge and the domain knowledge of different programming languages, and the classification layer calculates a prediction result of the masked language;

[0011] An instance x in the code corpus is acquired d The code pre-training model based on the Adapter network is trained to obtain the Adapter network with different programming language knowledge.

[0012] The low-resource programming language model is constructed, including an input layer, an encoding layer, a pooling layer and a classification layer, the input layer inputs an instance in the code corpus, the encoding layer calculates a semantic representation of the instance in the code corpus, the encoding layer is initialized by the pre-trained Adapter network corresponding to the programming language knowledge, the pooling layer calculates a semantic vector representation of the instance, and the classification layer calculates a prediction result for the covered word.

[0013] An instance x in the code corpus is acquired c The corresponding positive example and negative example are constructed respectively, and the low-resource programming language model is trained by using a classification cost and a contrast learning cost.

[0014] The final code pre-training model is trained, and the code pre-training result is output based on the final code pre-training model.

[0015] As a preferred technical solution, the general knowledge of the programming language includes data structures and algorithms, and programming paradigms and control flow structures.

[0016] As a preferred technical solution, an instance x in the code corpus is acquired d The code pre-training model based on the Adapter network is trained, and specifically includes:

[0017] The Transformer layer is initialized based on parameters of the code pre-training model CodeBERT;

[0018] The training instance x is input d The corresponding output is obtained through the plurality of Transformer layers, and is specifically represented as:

[0019]

[0020] Wherein, and respectively represent the kth and 12th Transformer layers of the code pre-training model CodeBERT, and represents the output corresponding to the kth and 12th Transformer layer of the code pre-training model CodeBERT;

[0021] Each layer of the Adapter network includes a multi-layer feedforward neural network and a Transformer layer;

[0022] Given a programming language p and an instance x d the initial semantic matrix representation of x and the output of the 1st Transformer layer of the code pre-training model CodeBERT The Adapter network is transformed as follows:

[0023]

[0024] where FFN1, FFN2 and FFNn represent the multi-layer feedforward neural network in the 1st, 2nd and N p Adapter layer, respectively, T1, T2 and TN are the Transformer layer in the 1st, 2nd and N p Adapter layer, respectively, and h1, h2 and hn are the output of the 1st, 2nd and N p Adapter layer, respectively, and [ ] represents the matrix concatenation operation.

[0025] The fusion layer includes a concatenation layer and an MLP layer, and the fusion layer calculates its final semantic vector representation h d,p as follows:

[0026]

[0027] where [ ] represents the matrix concatenation operation of the concatenation layer, and MLP represents the MLP layer.

[0028] The classification layer includes a fully connected layer and a softmax transformation, and obtains the final semantic vector representation h d of an instance x in the code corpus of a given programming language p. d,p After that, the classification layer calculates the prediction result of the mask language which is specifically represented as:

[0029]

[0030] where W1 and b1 are the parameters in the fully connected layer, is a v-dimensional vector, and v represents the size of the vocabulary during pre-training.

[0031] As a preferred technical solution, when training the code pre-training model based on the Adapter network, given the code corpus D c (x d ,y d ), the cross-entropy cost function of the mask language model is defined as follows:

[0032]

[0033] Where, θ c represents the parameter set of the entire model, represents the expected value of the prediction result with respect to the true label.

[0034] As a preferred technical solution, the encoding layer calculates the semantic representation of the instance in the code corpus, which is specifically represented as:

[0035] H c =Adapter p (x c )

[0036]

[0037] Where, H c , and are the semantic matrix representations of the instance x c , the positive example and the negative example , respectively, each column in the matrix corresponds to a word in the instance, Adapter p represents the encoding layer;

[0038] The pooling layer calculates the semantic vector representation of the instance, which is specifically:

[0039] h c =AvgPooling(H c )

[0040]

[0041] Where, h c , and are the semantic vector representations of the instance x c , the positive example and the negative example , respectively, and AvgPooling is the average pooling operation.

[0042] The classification layer includes a fully connected layer and a softmax transformation, and the semantic vector representation h c of the instance x cAfter that, the prediction result of the covered word is calculated by the classification layer of the mask language model Specifically represented as:

[0043]

[0044] where W2 and b2 are parameters of the full connection layer, is a v-dimensional vector, v is the size of the vocabulary in the corpus.

[0045] As a preferred technical solution, the classification cost adopts a cross-entropy cost function, which is specifically represented as:

[0046]

[0047] where θ' c represents the parameter set of the conjunction word classification model, y c is the real category to which the mask word in the instance x c belongs, represents the expected value of the category of the predicted word with respect to the category of the real word.

[0048] As a preferred technical solution, the contrast learning cost is represented as:

[0049]

[0050] where h c , and are semantic vector representations of the instance x c , the positive example and the negative example , sim() is used to measure the cosine distance between two vectors, || || represents the 2-norm of the vector, and T represents the transpose of the vector.

[0051] As a preferred technical solution, the total cost of the model is obtained by linearly summing the classification cost and the contrast learning cost, which is represented as:

[0052] L(θ' c ) = L MLM (θ' c ) + λL cl (θ' c )

[0053] where λ is a weight coefficient, used to adjust the importance of the pre-training cost and the contrast learning cost, L MLM (θ' c ) represents the classification cost, and L cl (θ' c ) represents the contrast learning cost.

[0054] The application also provides a code pre-training system based on an Adapter network and contrast learning, comprising an Adapter network-based code pre-training model construction module, an Adapter network-based code pre-training model training module, a low-resource programming language model construction module, a low-resource programming language model training module, and a code pre-training result output module.

[0055] The Adapter network-based code pre-training model construction module is configured to construct an Adapter network-based code pre-training model, comprising a plurality of Transformer layers, Adapter networks corresponding to different programming languages, a fusion layer, and a classification layer.

[0056] The Adapter network-based code pre-training model training module is configured to obtain training instances x d from a code corpus, train an Adapter network-based code pre-training model, and obtain an Adapter network with different programming language knowledge.

[0057] The low-resource programming language model construction module is configured to construct a low-resource programming language model, comprising an input layer, an encoding layer, a pooling layer, and a classification layer.

[0058] The low-resource programming language model training module is configured to obtain instances x c from a code corpus, construct corresponding positive examples and negative examples, and jointly train a low-resource programming language model using a classification cost and a contrast learning cost.

[0059] The code pre-training result output module is configured to output a code pre-training result based on a final code pre-training model.

[0060] The application also provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the above-mentioned code pre-training method based on an Adapter network and contrast learning.

[0061] Compared with the prior art, the application has the following advantages and beneficial effects:

[0062] The application constructs a code pre-training model based on an Adapter network, trains an Adapter network with different programming language knowledge based on a large amount of different programming language corpus, can reduce the problem of catastrophic forgetting of existing knowledge while pre-training code corpus of multiple programming languages, and supports training of new programming languages; constructs positive and negative examples for low-resource programming language data, jointly trains a low-resource programming language model using classification cost and contrastive learning cost, and can enhance the model capability under a low-resource programming language. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 The figure is a schematic diagram of the overall implementation architecture of the code pre-training method based on the Adapter network and contrastive learning of the application.

[0064] Figure 2 The figure is a schematic diagram of the flow of the code pre-training method based on the Adapter network and contrastive learning of the application.

[0065] Figure 3 The figure is a schematic diagram of the overall network architecture of the code pre-training model based on the Adapter network of the application.

[0066] Figure 4 The figure is a schematic diagram of the implementation process of the low-resource programming language model enhancement stage of the application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical scheme and advantages of the application clearer, the application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the application and do not limit the application.

[0068] Example 1

[0069] As shown in the figure, the present embodiment provides a code pre-training method based on an Adapter network and contrastive learning, which is divided into two stages: Figure 1

[0070] ​1) Multi-programming language code pre-training stage: This stage is based on a large amount of different programming language corpus, and a code pre-training model based on an Adapter network is constructed, which designs a general Transformer layer as a backbone network for each code language, initializes using an existing code pre-training model, and learns general knowledge. An Adapter network is constructed for each programming language to learn the domain knowledge of different programming languages, thereby maintaining the independence of the knowledge of each programming language, reducing the problem of catastrophic forgetting, and facilitating the subsequent pre-training of new programming languages. After training the model based on the Adapter network, suppose there is a new programming language to support, such as C language, only the Adapter network of C language needs to be added, and the Adapter network of C language is trained using C language code corpus data, and other code domains will not be affected.

[0071] 2) Low-resource programming language enhancement stage: This stage aims to enhance the Adapter of the low-resource programming language pre-trained in the previous stage. Specifically, it constructs positive and negative examples for the low-resource programming language data, uses contrastive learning to close the distance between positive examples and push away the distance between negative examples, and learns better code representation of low-resource programming languages, thereby enhancing the ability of the Adapter network of the low-resource programming language. The present invention not only avoids the problem of catastrophic forgetting of existing knowledge, but also supports the training of new programming languages at a very small training cost, and enhances the ability of the model on low-resource programming languages.

[0072] As shown in Figure 2 , the method specifically comprises the following steps:

[0073] S1: Construct a code pre-training model based on an Adapter network, which includes a plurality of Transformer layers, an Adapter network corresponding to different programming languages, a fusion layer, and a classification layer.

[0074] Specifically, the plurality of Transformer layers are used to learn the general knowledge of all programming languages. In this embodiment, 12 Transformer layers are preferred. Since the data of all programming languages will pass through the Transformer layer, the parameters of the Transformer layer are updated based on the data of all programming languages. Parameter updating allows the model to learn knowledge, so the knowledge of all programming languages is integrated into this Transformer layer.

[0075] The Adapter layer specific to different programming languages is used to learn the domain knowledge of the current programming language. Since the Adapter layer is only updated for the corresponding programming language data, the training is for the domain knowledge of the corresponding programming language.

[0076] The fusion layer is used to fuse the general knowledge and the domain knowledge of different programming languages;

[0077] The classification layer is used to calculate the prediction result of the mask language;

[0078] In this embodiment, the general knowledge of the programming language refers to some basic concepts and knowledge that all programming languages have, such as data structures and algorithms, programming paradigms, control flow structures, and the like, which are as follows:

[0079] Data structures and algorithms: data structures: array, linked list, stack, queue, tree, graph, hash table, etc. Algorithms: sorting, searching, recursion, dynamic programming, graph algorithm, greedy algorithm, etc.

[0080] Programming paradigms: for example, Java, C++, and Python are all object-oriented programming languages.

[0081] Control flow structure: for example, all languages have if-else similar control statements;

[0082] S2: as shown in Figure 3 , obtain a training instance x d , train the code pre-training model based on the Adapter network to obtain an Adapter network with different programming language knowledge, which specifically includes:

[0083] The Transformer layer is initialized based on the parameters of the code pre-training model CodeBERT, and the parameters remain unchanged in the following training;

[0084] Input the training instance x d , and obtain the corresponding output through multiple Transformer layers, which is specifically represented as:

[0085]

[0086] Wherein, and represent the kth and 12th Transformer layers of the code pre-training model CodeBERT, and represent the outputs corresponding to the kth and 12th Transformer layers of the code pre-training model CodeBERT, which can be used as the initial semantic matrix representation of the training instance x d , and k is a hyperparameter of the model, which can be set artificially;

[0087] In this embodiment, the training instance x dIt can be a piece of code, such as python code def[mask word]():print("Hello"), the model needs to predict what this mask word should fill in, such as predicting hello;

[0088] For different programming languages, the Adapter network can set different Adapter layers, the more code corpus of the programming language, the more knowledge it contains, and more parameters of the Adapter layer are needed to learn, and the design of the adaptive Adapter layer can also allocate the best Adapter layer number for each programming language, so as to achieve better results with as few parameters as possible;

[0089] For programming language p, set N p Adapter layers, N p is a hyperparameter of the Adapter network, which can be artificially specified according to the number of code corpus of the programming language in practical application, such as pyton has 10000 data, java 5000, and c++ 5000, and the proportion of total data is 0.5, 0.25, and 0.25. Artificially define a maximum Adapter layer, such as 10 layers, then the layer number of the three languages can be set to 0.5*10=5 layers, 0.25*10=2.5 (rounded up, can be set to 3 layers).

[0090] In this embodiment, each layer of the Adapter network is composed of a multi-layer feedforward neural network (FFN) and a Transformer layer, and the Adapter network takes the output of the first Transformer layer of the code pre-training model CodeBERT and the initial semantic matrix representation of the Adapter network as input, and further transforms these semantic matrix representations (features) to obtain semantic matrix representations (features) more suitable for each programming language. Compared with directly adjusting the parameters in the pre-trained encoding layer CodeBERT, the Adapter network adjusts the pre-trained model from another angle, so that it can be well used for pre-training tasks of different programming languages.

[0091] Specifically, given the programming language p and the initial semantic matrix representation d of instance x and the output of the first Transformer layer of the code pre-training model CodeBERT The Adapter network transforms as follows:

[0092]

[0093] FFN1, FFN2, and FFN N respectively represent the first, second, and Nth p multi-layer feedforward neural networks in the Adapter layer, respectively represent the first, second, and Nth p Transformer layers in the Adapter layer, and respectively represent the outputs of the first, second, and Nth p Adapter layers, and [] represents the matrix concatenation operation, is the initial input of the Adapter network, which can be randomly initialized or initialized as a zero matrix. The multi-layer feedforward neural network transforms the input vector to a smaller dimension, so that the parameters in the Adapter network are relatively small. The final output of the Adapter network is which can be regarded as the semantic matrix representation of the instance x d after transformation.

[0094] In this embodiment, the programming language p includes but is not limited to Java, C++, Python, and ruby;

[0095] In this embodiment, the knowledge of different programming languages is focused on learning the knowledge of their own programming languages through different Adapter layers, and the common knowledge existing between different programming language knowledge is learned through shared Transformer layers, so as to avoid the problem of catastrophic forgetting, and quickly support the pre-training of new programming languages. Removing the Adapter network of a programming language will not affect other programming languages.

[0096] After obtaining the semantic matrix representation of the instance x d after transformation and the semantic matrix representation of the instance x d,p after transformation , the two representations are input to the fusion layer for fusion. The fusion layer includes a concatenation layer and an MLP layer. Specifically, the fusion layer calculates the final semantic vector representation h d of the instance x d,p in the code corpus of the given programming language p in the following manner:

[0097]

[0098] wherein [] represents the matrix concatenation operation of the concatenation layer, and MLP represents the MLP layer.

[0099] The classification layer is composed of a fully connected layer and a softmax transformation, and obtains the final semantic vector representation h d of the instance x d,p in the code corpus of the given programming language p.After that, the classification layer calculates the prediction result of the mask language As shown below:

[0100]

[0101] where W1 and b1 are parameters in the fully connected layer, is a v-dimensional vector, v represents the size of the vocabulary during pre-training.

[0102] During training, given the code corpus D c instances (x d , y d ), the cross-entropy cost function of the mask language model is defined as follows:

[0103]

[0104] where θ c represents the parameter set of the entire model, represents the class of all masked words in instance x d , and represents the expected value of the prediction result with respect to the true label. Mask language refers to the way the model is trained, which is a masked way, such as instance x d is a piece of python code "def[masked word]():print(“hello”)", and the classification layer predicts the corresponding masked word, which ideally outputs "hello". After entering the adapter network of python, the output of the classification layer is a probability of various words in the vocabulary, and the word with the maximum probability is taken as the word corresponding to the masked word. If the classification layer predicts incorrectly, a loss will be generated, and under the drive of the loss, continuous training will make the prediction of the model more and more accurate, approaching the prediction of "hello".

[0105] After the code pre-training model based on the adapter network is pre-trained on a large amount of code corpus of programming languages, an adapter network with different programming language knowledge is obtained;

[0106] S3: Construct a low-resource programming language model, which includes an input layer, an encoding layer, a pooling layer, and a classification layer;

[0107] S4: As shown below, the low-resource programming language model is jointly trained using classification cost and contrastive learning cost; Figure 4

[0108] ​For some low-resource programming languages, the code corpus that can be obtained is very small, so compared with the high-resource Adapter network trained with a large amount of data, the Adapter network of the low-resource programming language is weak and it is difficult to learn enough knowledge of the low-resource programming language. The embodiment constructs the positive example (similar instance) and the negative example (dissimilar instance) of the given instance, and through learning, the distance of the model in the semantic space is closer to the positive example and farther from the negative example, so that the recognition ability of the model for the domain knowledge is stronger.

[0109] First, the positive and negative samples of the low-resource programming language are mined through data enhancement. In the input layer, for the instance x c in the given code corpus, and the negative example

[0110] For the construction of the positive example , unlike natural language, subtle changes in the code can greatly change the original meaning of the code, so the code is enhanced based on the semantic invariance rule. Specifically, first, the abstract syntax tree of the code is extracted, a variable (user-defined variable when writing program) in the abstract syntax tree is randomly named, and other variables corresponding to the variable in the whole code are also named in the same way, so that the function and semantics of the whole code are consistent. Then, the loop statement is rewritten, such as rewriting the for loop into a while statement with the same semantics, which can also ensure the semantic consistency of the code. Based on the above two kinds of semantic consistent data enhancement, multiple forms of positive examples can be generated for the instance x c .

[0111] The construction of the negative example can be divided into two cases: other instances different from the given instance x c can be selected as general negative examples; some more difficult to distinguish instances, such as instances with similar functions, can be designed as high-quality negative examples.

[0112] The instance x c , its positive example and negative example are combined into a training instance, denoted as

[0113] The given low-resource code instance x c , its positive example and negative example coding layer Adapter p is used to calculate the semantic representation of these instances. For the low-resource programming language p, the coding layer Adapter pinitialized by a pre-trained Adapter network corresponding to the programming language knowledge, specifically, input instance x c , its positive example and negative example goes through the encoding layer Adapter p and gets:

[0114] H c = Adapter p (x c )

[0115]

[0116] where H c , and are semantic matrix representations of instance x c , positive example and negative example , respectively, each column in the matrix corresponds to a word in the instance.

[0117] The pooling layer only includes an average pooling operation, which calculates the semantic vector representation of the instance, as follows:

[0118] h c = AvgPooling(H c )

[0119]

[0120] where h c , and are semantic vector representations of instance x c , positive example and negative example , respectively, and AvgPooling is the average pooling operation.

[0121] The classification layer consists of a fully connected layer and a softmax transformation, after obtaining the semantic vector representation h c of instance x c , the classification layer of the Masked Language Model (MLM) can calculate the prediction result for the covered words as follows:

[0122]

[0123] where W2 and b2 are the parameters of the fully connected layer, is a v-dimensional vector, and v is the size of the vocabulary in the corpus.

[0124] Specifically, given a code corpus D ctraining instances in The cross-entropy cost function is defined as follows:

[0125]

[0126] where θ' c represents the parameter set of the connecting word classification model, y c is the real class to which the mask word in the instance x c belongs, E yc [] represents the expected value of the class of the predicted word with respect to the class of the real word.

[0127] In addition to using the cross-entropy classification cost, a cost based on contrastive learning is also used, and the contrastive learning cost function is further defined as follows:

[0128]

[0129] where h c , and are the semantic vector representations of the instance x c , the positive example , and the negative example , sim() is used to measure the cosine distance between two vectors, || || represents the 2-norm of the vector, T represents the transpose of the vector, and minimizing the contrastive learning cost can make the distance between the instance and its positive example in the semantic space closer, while the distance between the instance and the negative example is farther, which is beneficial for distinguishing codes with similar semantics. Finally, the total cost of the model is the linear sum of the classification cost and the contrastive learning cost, as follows:

[0130] L(θ' c ) = L MLM (θ' c ) + λL cl (θ' c )

[0131] where λ is a weight coefficient used to adjust the importance of the pre-training cost and the contrastive learning cost.

[0132] The low-resource programming language model enhancement method based on contrastive learning can further enhance the effect of the low-resource programming language Adapter network. The classification layer predicts the probability of different words, and the classification cost calculates the difference between the predicted word and the real word. The goal of the contrastive learning cost is to make the code example and the positive example more similar and the negative example less similar. Through such training, the low-resource Adapter is equivalent to training more data (because one code can generate multiple positive examples, which will generate several more training data), and it can also distinguish different low-resource codes, which is equivalent to enhancing the Adapter's understanding of such low-resource.

[0133] S5: training a final code pre-training model, and outputting a code pre-training result based on the final code pre-training model.

[0134] Embodiment 2

[0135] The embodiment provides a code pre-training system based on an Adapter network and contrastive learning, which is used to implement the code pre-training method based on the Adapter network and the contrastive learning in Embodiment 1, and includes a code pre-training model construction module based on the Adapter network, a code pre-training model training module based on the Adapter network, a low-resource programming language model construction module, a low-resource programming language model training module, and a code pre-training result output module.

[0136] In the embodiment, the code pre-training model construction module based on the Adapter network is used to construct a code pre-training model based on the Adapter network, and includes a plurality of Transformer layers, Adapter networks corresponding to different programming languages, a fusion layer, and a classification layer. The plurality of Transformer layers learn general knowledge of programming languages, the Adapter networks learn domain knowledge of the current programming language, the fusion layer fuses the general knowledge and the domain knowledge of different programming languages, and the classification layer calculates a prediction result of a mask language.

[0137] In the embodiment, the code pre-training model training module based on the Adapter network is used to obtain training instances x d in a code corpus, train the code pre-training model based on the Adapter network, and obtain Adapter networks with different programming language knowledge.

[0138] In the embodiment, the low-resource programming language model construction module is used to construct a low-resource programming language model, which includes an input layer, an encoding layer, a pooling layer, and a classification layer. The input layer is used to input instances in the code corpus, the encoding layer calculates semantic representations of the instances in the code corpus, the encoding layer is initialized by a pre-trained Adapter network corresponding to programming language knowledge, the pooling layer calculates semantic vector representations of the instances, and the classification layer calculates a prediction result for a covered word.

[0139] In the embodiment, the low-resource programming language model training module is used to obtain instances x c in the code corpus, construct corresponding positive examples and negative examples respectively, and jointly train the low-resource programming language model by using a classification cost and a contrastive learning cost.

[0140] In the embodiment, the code pre-training result output module is used to output a code pre-training result based on a final code pre-training model.

[0141] Embodiment 3

[0142] The embodiment provides a storage medium, which can be a ROM, a RAM, a magnetic disk, an optical disk or the like storage medium, and the storage medium stores one or more programs, and the programs are executed by a processor to implement the code pre-training method based on an Adapter network and contrastive learning in the embodiment 1.

[0143] The above embodiment is a preferred embodiment of the present application, but the embodiments of the present application are not limited to the above embodiment, and any change, modification, substitution, combination, simplification made without departing from the spirit and principle of the present application should be an equivalent replacement mode, and all are included in the protection scope of the present application.

Claims

1. A code pre-training method based on Adapter networks and contrastive learning, characterized in that, Includes the following steps: Construct a code pre-trained model based on Adapter network, including: multiple Transformer layers, Adapter networks corresponding to different programming languages, fusion layer and classification layer. Multiple Transformer layers learn general knowledge of programming languages, Adapter network learns domain knowledge of the current programming language, fusion layer integrates general knowledge and domain knowledge of different programming languages, and classification layer calculates the prediction results of masked language. Depending on the amount of code corpus for the programming language, the number of Adapter layers in the Adapter network is set differently for different programming languages; Obtain training instance x from the code corpus d Train a code pre-trained model based on the Adapter network to obtain an Adapter network with knowledge of different programming languages, specifically including: The Transformer layer is initialized based on the parameters of the pre-trained code model CodeBERT; Given a programming language p and an instance x d Initial semantic matrix representation The output of the first Transformer layer of the pre-trained code model CodeBERT The Adapter network is transformed as follows: Among them, FFN1, FFN2 and They represent the 1st, 2nd, and Nth respectively. p A multi-layer feedforward neural network in an Adapter layer They are the 1st, 2nd, and Nth, respectively. p Transformer layer in an Adapter layer and They are the 1st, 2nd, and Nth, respectively. p The output of each Adapter layer, [] represents matrix concatenation operation; A low-resource programming language model is constructed, including an input layer, an encoding layer, a pooling layer, and a classification layer. The input layer takes instances from the code corpus as input, the encoding layer calculates the semantic representation of the instances from the code corpus, the encoding layer is initialized by a pre-trained Adapter network with corresponding programming language knowledge, the pooling layer calculates the semantic vector representation of the instances, and the classification layer calculates the prediction results for the masked words. The coding layer calculates the semantic representation of instances in the code corpus, specifically as follows: H c =Adapter p (x c ) Among them, H c , and For example, x c , positive example and negative examples The semantic matrix representation, where each column of the matrix corresponds to a word in the instance, Adapter p Indicates the coding layer; Obtain instance x from the code corpus c Construct corresponding positive and negative examples respectively, and jointly train the low-resource programming language model using classification cost and contrastive learning cost; For positive examples The construction of the code is based on semantic invariance rules to perform data augmentation on the code. First, the abstract syntax tree of the code is extracted, a variable in the abstract syntax tree is randomly named, and the corresponding variables in other positions in the whole code are also named in the same way. Loop statements are rewritten. The final code pre-trained model is obtained through training, and the code pre-training results are output based on the final code pre-trained model.

2. The code pre-training method based on Adapter network and contrastive learning according to claim 1, characterized in that, The general knowledge of the programming language includes: data structures and algorithms, as well as programming paradigms and control flow structures.

3. The code pre-training method based on Adapter network and contrastive learning according to claim 1, characterized in that, Obtain training instance x from the code corpus d Training a code pre-trained model based on the Adapter network specifically includes: Input training instance x d The corresponding output is obtained after passing through multiple Transformer layers, specifically as follows: in, and These represent the k-th and 12th Transformer layers of the pre-trained CodeBERT model, respectively. and This represents the outputs of the k-th and 12th Transformer layers of the pre-trained CodeBERT model; Each layer of the Adapter network consists of a multi-layer feedforward neural network and a Transformer layer; The fusion layer consists of a concatenation layer and an MLP layer. The fusion layer calculates its final semantic vector representation h as follows: d,p : Where [] represents the matrix concatenation operation of the concatenation layer, and MLP represents the MLP layer; The classification layer consists of a fully connected layer and a softmax transform to obtain instances x in a code corpus of a given programming language p. d The final semantic vector representation h d,p Then, the classification layer calculates the prediction results of the masked language. Specifically, it is expressed as follows: Where W1 and b1 are parameters in the fully connected layer, It is a v-dimensional vector, where v represents the size of the vocabulary during pre-training.

4. The code pre-training method based on Adapter network and contrastive learning according to claim 3, characterized in that, When training a code pre-training model based on an Adapter network, given a code corpus D... c Instances (x) d ,y d The cross-entropy cost function of the masked language model is defined as follows: Where, θ c This represents the parameter set of the entire model. This represents the expected value of the prediction result with respect to the true label.

5. The code pre-training method based on Adapter network and contrastive learning according to claim 1, characterized in that, The semantic vector representation of pooling layer computation instances is as follows: h c =AvgPooling(H c ) Among them, h c , and For example, x c , positive example and negative examples The semantic vector representation, where Avgpooling is the average pooling operation; The classification layer consists of a fully connected layer and a softmax transform, which is used to obtain instance x. c semantic vector representation h c Then, the classification layer of the masked language model calculates the prediction results for the masked words. Specifically, it is expressed as follows: Where W2 and b2 are the parameters of the fully connected layer, It is a v-dimensional vector, where v is the size of the vocabulary in the corpus.

6. The code pre-training method based on Adapter network and contrastive learning according to claim 5, characterized in that, The classification cost uses the cross-entropy cost function, which is specifically expressed as: Where, θ′ c y represents the parameter set of the connective word classification model. c For instance x c The true category to which the masked word belongs. This represents the expected value of the predicted word's category with respect to the actual word's category.

7. The code pre-training method based on Adapter network and contrastive learning according to claim 5, characterized in that, The cost of contrastive learning is expressed as: Among them, h c , and For example, x c , positive example and negative examples The semantic vector representation of sim() is used to measure the cosine distance between two vectors, ‖‖ represents the 2-norm of the vector, and T represents the transpose of the vector.

8. The code pre-training method based on Adapter network and contrastive learning according to claim 5, characterized in that, The total cost of the model is obtained by linearly summing the classification cost and the contrastive learning cost, and is expressed as: L(θ′ c )=L MLM (θ′ c )+λL cl (θ′ c ) Where λ is a weighting coefficient used to adjust the importance of pre-training cost and contrastive learning cost, L MLM (θ′ c ) represents the classification cost, L cl (θ′ c ) indicates the cost of comparative learning.

9. A code pre-training system based on Adapter networks and contrastive learning, characterized in that, The code pre-training method based on Adapter network and contrastive learning as described in any one of claims 1-8 includes: a code pre-training model construction module based on Adapter network, a code pre-training model training module based on Adapter network, a low-resource programming language model construction module, a low-resource programming language model training module, and a code pre-training result output module. The code pre-training model building module based on the Adapter network is used to build a code pre-training model based on the Adapter network, including: multiple Transformer layers, Adapter networks corresponding to different programming languages, fusion layer and classification layer. Multiple Transformer layers learn the general knowledge of the programming language, the Adapter network learns the domain knowledge of the current programming language, the fusion layer integrates the general knowledge and the domain knowledge of different programming languages, and the classification layer calculates the prediction results of the masked language. The code pre-training model training module based on the Adapter network is used to obtain training instances x from the code corpus. d Train a code pre-trained model based on the Adapter network to obtain an Adapter network with knowledge of different programming languages; The low-resource programming language model building module is used to build a low-resource programming language model, including an input layer, an encoding layer, a pooling layer, and a classification layer. The input layer takes instances from the code corpus as input, the encoding layer calculates the semantic representation of the instances from the code corpus, the encoding layer is initialized by a pre-trained Adapter network with corresponding programming language knowledge, the pooling layer calculates the semantic vector representation of the instances, and the classification layer calculates the prediction results for the masked words. The low-resource programming language model training module is used to obtain instance x from the code corpus. c Construct corresponding positive and negative examples respectively, and jointly train the low-resource programming language model using classification cost and contrastive learning cost; The code pre-training result output module is used to output the code pre-training result based on the final code pre-training model.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the code pre-training method based on Adapter network and contrastive learning as described in any one of claims 1-8.

Images