System and Method for Encoder-Decoder-Based Frameworks for Code Generation and Understanding

Abstract

The embodiments described herein provide a hybrid encoder-decoder Transformer framework for multitask pre-training and flexible fine-tuning for both code comprehension and generation tasks. Specifically, the framework is built on multimodal encoder and decoder modules. During pre-training, the encoder-decoder framework is trained for multiple learning objectives, including a diverse set of self-supervised tasks across two main stages: pre-training on unimodal and bimodal data.

Description

Technical Field

[0001] [Cross - Reference] This international application claims the priority of U.S. Patent Application No. 18 / 174,547, which was filed on February 24, 2023, is co - pending and jointly owned, and this U.S. Patent Application is hereby expressly incorporated by reference in its entirety.

[0002] This application relates to U.S. Patent Application No. 18 / 475,103, which was filed on September 26, 2023, is co - pending and jointly owned, and U.S. Patent Application No. 17 / 459,968, which was filed on August 27, 2021, and U.S. Provisional Application No. 63 / 189,857, which was filed on May 18, 2021, and these are hereby expressly incorporated by reference in their entirety.

[0003] [Technical Field to Which the Invention Belongs] Embodiments generally relate to machine learning systems for code - related tasks, and more specifically, to encoder - decoder - based Transformer networks for code generation and understanding.

Background Art

[0004] Machine learning systems are widely used in a variety of natural language processing and / or code-related tasks. For example, large-scale language models (LLMs) are employed to pre-train on source code data for various downstream tasks in the code domain, such as code generation and comprehension tasks. By pre-training LLMs on large amounts of codebase data (e.g., GitHub public data), these LLMs can learn rich contextual representations that can be passed on to the relevant downstream code-related tasks. However, existing models are often designed to work well only for a subset of tasks (e.g., generation-only tasks or comprehension-only tasks). For example, encoder-only models are often used to perform comprehension tasks such as text-to-code search. For generation tasks such as code generation, decoder-only models are often used.

[0005] Therefore, a code generation framework is needed that can adapt to multiple types of code-related tasks. [Brief explanation of the drawing]

[0006] [Figure 1] This is a simplified diagram illustrating an encoder-decoder pre-training framework that includes two pre-training stages for code generation and understanding, according to several embodiments. [Figure 2] This is a simplified diagram illustrating an exemplary embodiment of a stage 1 unimodal pre-training for code-only data according to one or more embodiments described herein. [Figure 3A] This is a simplified diagram illustrating an exemplary embodiment of the Stage 2 bimodal pre-training on text-code pair data according to one or more embodiments described herein. [Figure 3B] This is a simplified diagram illustrating an exemplary structure of an encoder-decoder model in an alternative embodiment according to one or more embodiments described herein. [Figure 4] This is a simplified block diagram showing the inference or fine-tuning stage of an encoder-decoder model pre-trained through the pre-training stages shown in Figures 1 to 3, according to one or more embodiments. [Figure 5] This is a simplified block diagram illustrating an integrated search extension generation paradigm according to one or more embodiments described herein. [Figure 6] This is a simplified diagram showing a computing device implementing the encoder-decoder model described in Figures 1 to 3B, according to one embodiment described herein. [Figure 7] Figures 1 to 6 are simplified block diagrams of networked systems suitable for implementing the encoder-decoder-based code understanding and generation framework, as well as other embodiments described herein. [Figure 8] This is an exemplary logic flow diagram illustrating how to train an encoder-decoder-based framework for code-related tasks based on the frameworks shown in Figures 1 to 7, according to some embodiments described herein. [Figure 9] This specification provides exemplary data tables showing the results of data experiments on the encoder-decoder pre-training framework described in Figures 1 to 8, according to embodiments described herein. [Figure 10] This specification provides exemplary data tables showing the results of data experiments on the encoder-decoder pre-training framework described in Figures 1 to 8, according to embodiments described herein. [Figure 11] This specification provides exemplary data tables showing the results of data experiments on the encoder-decoder pre-training framework described in Figures 1 to 8, according to embodiments described herein. [Figure 12] This specification provides exemplary data tables showing the results of data experiments on the encoder-decoder pre-training framework described in Figures 1 to 8, according to embodiments described herein. [Figure 13]This specification provides exemplary data tables showing the results of data experiments on the encoder-decoder pre-training framework described in Figures 1 to 8, according to embodiments described herein. [Figure 14] This specification provides exemplary data tables showing the results of data experiments on the encoder-decoder pre-training framework described in Figures 1 to 8, according to embodiments described herein.

[0007] Embodiments of this disclosure and their advantages are best understood by referring to the detailed description below. Similar reference numerals are used to identify similar elements shown in one or more drawings, and it should be understood that the illustrations in the drawings are for illustrative purposes only and not to limit the disclosure. [Modes for carrying out the invention]

[0008] As used herein, the term “Network” may include any hardware or software-based framework, including any artificial intelligence network or system, a neural network or system, and / or any training or learning model implemented on or with it.

[0009] As used herein, the term “module” may include a hardware or software-based framework that performs one or more functions. In some embodiments, a module may be implemented on one or more neural networks.

[0010] Existing large-scale language models (LLMs) are employed to pre-train on source code data for various downstream tasks in the code domain, such as code generation and comprehension tasks. Such existing LLMs often employ specific architectures that are limited to encoders only or decoders only for different downstream tasks, or they rely on a single network for all code-related tasks. The performance of the encoder-only or decoder-only paradigm is severely limited by its inflexibility in applying to different downstream tasks. However, the single-network-for-all paradigm requires fine-tuning and activation of a large set of all model parameters, making the training process time- and resource-intensive.

[0011] Given the need to provide a flexible framework for both code generation and comprehension, the embodiments described herein provide a hybrid encoder-decoder Transformer framework for multitask pre-training and flexible fine-tuning for both code comprehension and generation tasks. Specifically, the framework is built on multimodal encoder and decoder modules. During pre-training, the encoder-decoder framework is trained for multiple learning objectives, including a diverse set of self-supervised tasks across two main stages: pre-training on unimodal and bimodal data. For example, employing a stepwise pre-training strategy, the encoder-decoder framework is first trained on code-only data using span denoising and causal language modeling (CLM) tasks. Then, in the second training stage, the encoder-decoder framework is trained on text code data using cross-modal contrast learning, matching, and CLM tasks.

[0012] In one embodiment, the encoder-decoder framework includes a hybrid of encoder-decoder Transformers. For example, the encoder has a plurality of encoder sub-modules that operate in parallel and share the same parameters. The decoder may have a plurality of decoder sub-modules that operate in parallel and share the same parameters except for the last feed-forward layer (FFN) that functions as a decoder head adapted to different decoding tasks.

[0013] In one embodiment, the weight sharing strategy can be adopted through task-specific experts, i.e., the experts are designed for different learning tasks while receiving the same backbone context representation. In this way, in order to optimize multi-task learning while efficiently activating the correct model parameters, the experts can share the parameters from the trained encoder-decoder framework with the feed-forward decoder heads for a specific learning task. In the encoder-decoder Transformer structure, only one feed-forward layer expert for each task is activated.

[0014] The embodiments described herein provide several advantages. For example, the component modules can be separated / joined based on different training or application tasks, such as adaptation as an integrated search expansion generation system. In this way, the encoder-decoder framework can be adapted to various downstream tasks and functions. overview

[0015] FIG. 1 is a schematic diagram showing an overview of an encoder-decoder pre-training framework including two pre-training stages 100a and 100b for code generation and understanding according to some embodiments. An encoder-decoder model 110 including a plurality of Transformer-based encoders and decoders is used. Each of the Transformer-based encoders or decoders can operate by combining specific functions, and the combination functions as an integrated multi-task encoder-decoder model 110. Further details of the architecture of the encoder-decoder 110 may be described in connection with FIGS. 3A-3B.

[0016] In one embodiment, the encoder-decoder model 110 can be trained according to a set of self-supervised tasks over two pre-training stages: unimodal pre-training (stage 1 110a) and bimodal pre-training (stage 2 100b). In the first stage 100a of unimodal pre-training, the vanilla encoder-decoder Transformer model 110 is pre-trained with only a large amount of code data 102 using computationally efficient objectives such as span noise removal loss 112 and causal language modeling loss 114. Further details of the pre-training 110a in stage 1 will be described below in connection with FIG. 2.

[0017] In the second stage 100b of bimodal pre-training, the encoder-decoder model 110 inherits the previously trained encoder-decoder parameters 116 from the first stage 110a and serves as the backbone for initializing the mixed encoder-decoder Transformer. The encoder-decoder Transformer model 110 is then trained with a smaller set of code-text data 104 for cross-modal learning purposes, including text-code comparison loss 118, text-code matching loss 120, code-text generation loss 122, and text-code generation loss 124. Further details of the pre-training stage 2 110b are described below in relation to Figures 3A and 3B.

[0018] In this way, the stepwise training approach 100a-b efficiently exposes the encoder-decoder model 110 to a wider variety of data and thus learns rich contextual representations of code and text. For each step, multiple pre-training objectives (e.g., 112 and 114 in step 1 110a, and 118, 120, 122, and 124 in step 2 100b) can be optimized together with equal weights.

[0019] Figure 2 is a simplified diagram illustrating an exemplary embodiment of Stage 1 unimodal pre-training on code-only data according to one or more embodiments described herein. As shown in Figure 2, during Stage 1 pre-training 100a, the encoder-decoder model 110 is pre-trained on a large amount of code-only unimodal data 102 shown in Figure 1. Code-only unimodal data 102 can often be obtained from open-source platforms such as GitHub. The encoder-decoder model 110 is pre-trained from scratch (e.g., from a vanilla encoder-decoder model) using a combination of span denoising and causal language modeling (CLM) tasks.

[0020] In one embodiment, encoder input 202a and decoded output 202b illustrate an example of span-denoising training. For example, a portion of the code-only input (e.g., 15% of its tokens) may be randomly replaced by an indexed sentinel token (such as [MASK0]). The training input with [MASK0] can be encoded into a coded representation by the encoder of Model 110, similar to the code input 202a. The decoder of Model 110 is then configured to reconstruct the original unmasked code input by generating a combination of randomly replaced spans. An exemplary decoded output may be shown in 202b. The span-denoising loss 112 (shown in Figure 1) can then be calculated by comparing the output span 202b with the original unreplaced span in the code input, e.g., the cross-entropy loss.

[0021] In one embodiment, the training input 202a may be generated using whole-word masking by sampling the span before subword tokenization to avoid partial subtokens being masked.

[0022] In one embodiment, the encoder input 206a and the decoded output 206b illustrate an example of a modified CLM training for optimizing the encoder-decoder model 110 for autoregressive generation. For example, pivot positions may be randomly selected within the code-only input. In one embodiment, pivot positions are uniformly sampled between 10% and 90% of the entire code-only sequence.

[0023] In this way, the context before the pivot point can be treated as the source sequence, and the sequence after the pivot point can be treated as the target output. The encoder input 206a may contain special tokens [CLM] prepared in the source sequence and encoded into a code representation by the encoder of model 110. The decoder of model 110 then generates a predicted code sequence 206b next to the source sequence. The CLM loss (114 in Figure 1) can then be calculated by comparing the predicted code sequence 206b with the target sequence, e.g., the cross-entropy loss.

[0024] In one embodiment, encoder input 204a and decoder output 204b illustrate an example of another variation of CLM training. Specifically, the second CLM variation is a decoder-only generation task and can be considered an extreme case of the first variation. A single [CLM] token is passed as encoder input 204a, and therefore only a decoder is required to generate a complete code sequence 204b based on the encoded representation of [CLS]. The CLM loss for this variation (114 in Figure 1) is calculated similarly to that of the first variation. Compared to the first variation shown by encoder input 206a and decoder output 206b, this task aims to provide a higher density of training signals to train the decoder as an independent, full-fledged code generation module.

[0025] In one embodiment, by combining a span denoising task and a CLM task, the encoder-decoder model 110 may be updated via backpropagation based on the sum of the span denoising loss 112 and the CLM loss 114. In this way, the encoder-decoder model 110 learns to reconstruct the code context at different scales, namely, code span, partial program, and complete program.

[0026] Figure 3A is a simplified diagram illustrating an exemplary embodiment of the Stage 2 bimodal pre-training 100b for text-code pair data according to one or more embodiments described herein. Figure 3A shows an exemplary architecture in which the encoder and decoder components are initialized from checkpoints obtained from the first pre-training stage 100a. The encoder-decoder architecture comprises an encoder 110a and a decoder 110b.

[0027] Specifically, the encoder 110a may comprise one or more (e.g., two) bimodal encoder submodules 310a and 310b operating in parallel. These bimodal encoder submodules 310a and 310b may be identical and share the same parameters.

[0028] In the second stage 100b, the encoder-decoder model 110 is pre-trained using weights / parameters pre-trained from the first stage 100a, with text-code bimodal data (e.g., 104 in Figure 1) containing text-code pairs. Each text-code pair includes a code function 104b and its corresponding docstring 104a describing its semantics. Such a bimodal data format facilitates the exploration of model training for cross-modal understanding and generation.

[0029] In one embodiment, the bimodal encoder 310a receives a text input 104a and encodes it into a continuous text representation through bidirectional self-attention and feedforward layer processing. Similar to BERT, special tokens [CLS] are prepended to the text input, and the output embedding in the final Transformer layer of the bimodal encoder 310a becomes the corresponding representation 304a of the input text. A linear layer is added to map the output representation to a 256-dimensional vector, and L2 normalization is performed to obtain the text representation 304a.

[0030] In parallel with the bimodal encoder 310a, the bimodal encoder 310b receives the code snippet input 104b and encodes it into a continuous code representation via bidirectional self-attention and feedforward layer processing. Similar to BERT, special tokens [CLS] are prepended to the code input, and the output embedding in the final Transformer layer of the bimodal encoder 310b becomes the corresponding representation of the input text. A linear layer is added to map the output representation to a 256-dimensional vector, and L2 normalization is performed to obtain the code representation 304b.

[0031] In one embodiment, text representation 304a and code representation 304b may be used to compute a text-code contrast loss 118. Specifically, the text-code contrast learning task aligns the feature spaces of text and code encoders by combining representations of positive text-code pairs and separating negative pairs. This task activates only bimodal encoders 310a~b to generate text / code embeddings / representations 304a~b. For example, given text sample T104a and code sample C104b, the representation of text T ht304a and the representation of code C hc304b are generated as described above, for example, by mapping the [CLS] embedding to normalized low-dimensional (256-d) representations from bimodal encoders 310a~b.

[0032] Given a batch of N text-code pairs, to calculate text-code similarity and code-text similarity,

number

number

number

[0033] In one embodiment, to enrich the negative samples, a momentum encoder is employed in addition to the bimodal encoders 310a-b, and the embedding of samples 104a-b from previous minibatches can be stored. Specifically, the momentum encoder maintains a queuing system that enqueues samples in the current minibatch and dequeues samples in the oldest minibatch. To ensure consistency of representation between training steps, the momentum encoder can be updated by linear interpolation of the original encoder and the momentum encoder. Furthermore, since text samples and code samples may be loosely paired, and each text / code sample may have multiple positive pairs, the momentum encoder can create soft labels and consider potential positives among negative pairs. Further details of the momentum encoder are described in concurrently pending and co-owned U.S. Patent Nonprovisional Application No. 17 / 745,540 (Li et al., 2022), which is expressly incorporated herein by reference in its entirety.

[0034] On the decoder side, decoder 110b may have one or more (e.g., two, three, and / or four, etc.) decoder submodules 320a~d operating in parallel. These decoder submodules may be different decoders according to different training tasks, for example, a bimodal matching decoder 320a for a text code matching task, a unimodal generation decoder 320a or 320c for a generation task. Decoder submodules 320a~c may share similar structures and similar parameters, except for a final feedforward layer that acts as a respective decoder head adapted to produce their respective decoder outputs according to a specific training task.

[0035] In one embodiment, the bimodal matching decoder 320a may predict whether text 104a and code snippet 104b share the same semantics according to a text-code matching (TCM) task. This task aims to activate the bimodal matching decoder and learn a better bimodal representation that captures finer alignment between text and code modality.

[0036] Specifically, given a code sample 104b, a task-specific [Match] token is prepended in the code input sequence 104b to inform the decoder 320a of the text-to-code matching capability, and an [EOS] token is appended to the end of the code input 104b. The bimodal matching decoder 320a first passes the prepended code snippet to the embedding layer and the causal self-attention layer. The self-attention representation is then passed to the cross-attention layer, which queries the relevant signal from the text representation 304a (received from the bimodal encoder 310a). Since decoder 320a uses a causal self-attention mask and only the last decoder token is attentionable in all contexts, the output embedding of [EOS] in the last decoder layer is used as the text-to-code cross-modal alignment representation.

[0037] For the binary classification task, a linear layer is built on top of the output embedding of decoder 320a, which predicts whether a text-code pair is positive (matches) or negative (mismatches). The output embedding of the [EOS] token is used as a fused bimodal representation of the text-code pair (T,C). Following the linear layer and softmax, the two-class probability p is calculated. tcm (T) can be calculated, and therefore the TCM loss 120 can be calculated as follows:

number

[0038] In parallel with the bimodal matching decoder 320a, the unimodal generation decoders 320b-c can be used for training on a text-code dual generation task. This generation task focuses on cross-modal generation objectives between text and code via dual multimodal transformations: text-to-code generation and code-to-text generation (i.e., code summarization). Each transformation separately activates the corresponding (unimodal) code / text generation decoder 320b or 320c. This task can, for example, use text-to-code generation to bridge the gap between the pre-training and fine-tuning phases in a generation-based bimodal application task.

[0039] Specifically, the unimodal generation decoders 320b~c can generate output sequences in a programming language / natural language. These decoders follow the same design as the bimodal matching decoder 310a, which has a causal attention layer and a cross-attention layer. When the input is a text sample 104a to the encoder, the code generation decoder is used, and the code snippet 104b is prepended with the [CDec] token as the first token in the input sequence to decoder 320b. The code generation decoder 320b operates in code generation function to generate a predicted code sequence corresponding to the text input 104a. The predicted code sequence is then compared with the code sequence 104b to obtain the text-to-code generation loss 124 L t2c For example, calculate the cross-entropy loss.

[0040] If the input is a code sample 104b to the encoder, the text generation decoder 320c is used, and the [TDec] token is prepended to the text sample 104a as the input sequence to the decoder 320c. The text generation decoder 320c operates in its text generation (i.e., code summarization) function to generate predicted text corresponding to the code input 104b. The predicted text is then compared to the text sequence 104a to obtain the code-to-text generation loss 122 L.c2t For example, calculate the cross-entropy loss.

[0041] Thus, the complete pre-training loss for the second stage can be the sum of the losses:

number

[0042] Next, the encoder 110a and decoder 310b can be updated together via backpropagation using the pre-training loss from the second stage.

[0043] In one embodiment, when only a single encoder within encoder 110a is used to process bimodal data, a partial sharing scheme is employed among the three decoders 320a-c. Specifically, the parameters of the self-attention layer and cross-attention layer are shared among the decoders. Sharing contextual representations in these layers among the text-code matching decoder 320a and the text-code dual generation decoders 320b-c can enable generalization between tasks at the context level. In this way, these multimodal training tasks benefit from sharing contextual representations. For example, the text-code generation decoder 320b may benefit from the coded representation of semantic recognition jointly learned from the text-code matching task.

[0044] Figure 3B is a simplified diagram showing an exemplary structure of an encoder-decoder model 110 in an alternative embodiment according to one or more embodiments described herein. The CodeT5Mix mixed architecture can also be interpreted from another perspective as an integrated encoder-decoder having task-specific FFN experts within a shared decoder, as shown in Figure 3B. Note that while there may be an extra parameter cost of two FFN layers compared to a standard encoder-decoder model during pre-training, there is no additional computational cost when fine-tuning a particular downstream task, as only one FFN layer is activated.

[0045] Figure 4 is a simplified block diagram showing an inference or fine-tuning stage 400 of an encoder-decoder model pre-trained via the pre-training stages shown in Figures 1 to 3, according to one or more embodiments. In the inference or fine-tuning stage 400, depending on the type of downstream task, a pre-trained encoder-only model 110a, a decoder-only model 110b, or an encoder-decoder model 110 may be selected to perform a particular downstream task.

[0046] For example, for understanding tasks such as text-to-code search, defect detection, and clone detection, the understanding task input 420 can be passed to a pre-trained bimodal encoder 110a to obtain text / code embeddings, which can then be passed to a binary classifier for detection or search tasks. Furthermore, a pre-trained text-to-code matching decoder 110b (e.g., activation 320b in Figure 3A) can be added to predict matching probabilities.

[0047] In another example, for a decoder-only task, a [CLM] token is prepended to the encoder input, passing the source sequence to the decoder as a prefix context. Then, to reduce the number of trainable parameters, the weights of the encoder 110a and the cross-attention layer weights in the decoder 110b are frozen.

[0048] As another example, in the case of generative tasks such as code summarization, code completion, code generation, and mathematical programming, task-specific experts may be selected based on the output modality.

[0049] In one embodiment, feedforward layers (FFNs) 402, 404, and 406 with distinct weights may be deployed while sharing context representations between tasks. Specifically, FFN layers 402-406 function as task-specific experts in the encoder-decoder model 110 to reduce inter-task interference. For example, matching FFN 402, code generation FFN 404, or text generation FFN 406 may each function as task experts to receive context representations from the pre-trained decoder 110b in Figure 3A and share them among decoders according to a particular task.

[0050] Another advantage of this shared approach is that it efficiently activates the correct model parameters for different application tasks while keeping the model size manageable. While a weight-sharing scheme that fully shares all parameters between decoders can save more parameter costs, it should be noted that this can lead to significant interference and performance degradation in downstream tasks. Furthermore, only a single task expert (e.g., FFN402, 404, or 406 depending on the task) is activated for each downstream task, thus avoiding the extra parameter costs of task experts during fine-tuning, as no additional computational costs are incurred.

[0051] Figure 5 is a simplified block diagram showing an integrated search-enhancement-generation paradigm 500 according to one or more embodiments described herein. Paradigm 500 shows that a pre-trained encoder-decoder model 110 is used for both code retrieval and generation, and the results are then aggregated as an integrated semi-parametric search-enhancement-generation generator. For example, a text input 501 is passed to the encoder-decoder model 110, which provides a generated output 503 according to a code generation task and a search output 505 according to a retrieval task. The retrieved code 505 provides important context (e.g., using "urllib3" for an HTTP request) to guide the generation process for more accurate predictions, resulting in a post-retrieval-generated output 507. In contrast, the generation task gives an incorrect prediction; for example, the generated-only code 503 only captures the concepts of "download" and "compress". Computer and network environment

[0052] Figure 6 is a simplified diagram showing a computing device implementing the encoder-decoder model 110 described in Figures 1 to 3B, according to one embodiment described herein. As shown in Figure 6, the computing device 600 includes a processor 610 coupled to memory 620. The operation of the computing device 600 is controlled by the processor 610. Although the computing device 600 is shown to have only one processor 610, it is understood that the processor 610 may represent one or more central processing units, multicore processors, microprocessors, microcontrollers, digital signal processors, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), graphics processing units (GPUs), etc., within the computing device 600. The computing device 600 may be implemented as a standalone subsystem, as a board added to a computing device, and / or as a virtual machine.

[0053] Memory 620 may be used to store software executed by computing device 600 and / or one or more data structures used during the operation of computing device 600. Memory 620 may include one or more types of machine-readable media. Some common forms of machine-readable media may include floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0054] The processor 610 and / or memory 620 can be located in any suitable physical arrangement. In some embodiments, the processor 610 and / or memory 620 may be implemented on the same board, in the same package (e.g., system-in-package), on the same chip (e.g., system-on-chip), etc. In some embodiments, the processor 610 and / or memory 620 may include distributed, virtualized, and / or containerized computing resources. Consistent with such embodiments, the processor 610 and / or memory 620 may be located in one or more data centers and / or cloud computing facilities.

[0055] In some examples, memory 620 may include a non-temporary tangible machine-readable medium containing executable code that, when executed by one or more processors (e.g., processor 610), can cause one or more processors to perform methods further described herein. For example, as shown in the figure, memory 620 includes instructions for a code understanding and generation module 630 that can be used to implement and / or emulate systems and models, and / or to implement any of the methods further described herein. The code understanding and generation module 630 may receive input 640, such as input training data (e.g., code-only data 102 or code-text pairs 104), via a data interface 615, and generate output 650, which may be code or text output. An example of input data may be a code-only input sample. An example of output data may be a text output describing a code input sample according to a code understanding task.

[0056] The data interface 615 may include a communication interface and a user interface (such as a voice input interface or a graphical user interface). For example, the computing device 600 may receive input 640 (such as a training dataset) from a networked database via the communication interface. Alternatively, the computing device 600 may receive input 640, such as a code sample or a text sample, from a user via the user interface.

[0057] In some embodiments, the code understanding and generation module 630 is configured to generate code or text output according to specific code-related tasks such as code generation, code summarization, code retrieval, and code understanding. The code understanding and generation module 630 may further include an encoder submodule 631 (e.g., similar to 110a in Figure 3A), a decoding submodule (e.g., similar to 110b in Figure 3A), and a code task FFN submodule 633 (e.g., similar to FFNs 402, 404, and 406 in Figure 4). In one implementation, the code task FFN submodule 633 may be part of the decoder submodule 632, as shown in Figure 3B.

[0058] In one embodiment, the code understanding and generation module 630 and its submodules 631-633 may be implemented by hardware, software, and / or a combination thereof.

[0059] In one embodiment, one or more of the code comprehension and generation module 630 and its submodules 631 may be implemented via an artificial neural network. The neural network includes a computing system built on a collection of connected units or nodes called neurons. Each neuron receives an input signal and then generates an output by a nonlinear transformation of the input signal. Neurons are often connected by edges, and often these edges are associated with adjustable weights. Neurons are often aggregated into layers such that different layers perform different transformations on their respective inputs and output the transformed input data to the next layer. Thus, the neural network may be stored in memory 620 as parameters describing the structure of the layers of neurons, as well as the nonlinear transformations in each neuron and the weights associated with the edges connecting the neurons. An exemplary neural network may be a Transformer network, for example.

[0060] In one embodiment, one or more of the neural network-based code comprehension and generation module 630 and its submodules 631-633 may be trained by updating the underlying parameters of the neural network based on the loss described with respect to Figures 1-3B. For example, the loss described in equation (5) is a metric that evaluates how far the predicted output value generated by the neural network model is from its target output value (also called the “ground truth” value). Given the loss calculated according to equation (5), the negative gradient of the loss function is calculated individually for each weight of each layer. Such negative gradients are calculated iteratively in reverse, from the last layer of the neural network toward the input layer, one layer at a time. The parameters of the neural network are updated in reverse, from the last layer toward the input layer, based on the calculated negative gradient, in order to minimize the loss (backpropagation). Backpropagation from the last layer toward the input layer may be performed for several training samples in several training epochs. In this way, the parameters of the neural network can be updated in a direction that reduces or minimizes the loss, indicating that the neural network has been trained to produce predicted output values ​​that are closer to the target output value.

[0061] Some examples of computing devices such as computing device 600 include non-temporary, tangible, machine-readable media containing executable code that, when executed by one or more processors (e.g., processor 610), causes one or more processors to execute the process of the Method. Some common forms of machine-readable media that may contain the process of the Method include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, and / or any other media adapted to be read by a processor or computer.

[0062] Figure 7 is a simplified block diagram of a networked system 700 suitable for implementing the encoder-decoder-based code understanding and generation frameworks described in Figures 1 to 6, as well as other embodiments described herein. In one embodiment, the system 700 includes a user device 710 that can be operated by a user 740, data vendor servers 745, 770, and 780, a server 730, and other forms of devices, servers, and / or software components that operate to perform various methodologies according to the embodiments described. Exemplary devices and servers may include devices, standalone, and enterprise-class servers, which may be similar to the computing device 600 described in Figure 6, and run an OS such as MICROSOFT® OS, UNIX® OS, LINUX® OS, or other suitable device and / or server-based OS. The devices and / or servers shown in Figure 7 may be deployed in other ways, and the operations performed and / or services provided by such devices and / or servers may be combined or separated for a given embodiment, and may be performed by more or fewer devices and / or servers. One or more devices and / or servers may be operated and / or maintained by the same or different entities.

[0063] The user device 710, data vendor servers 745, 770, and 780, and server 730 can communicate with each other via network 760. The user device 710 is utilized by user 740 (e.g., driver, system administrator, etc.) and can access various functions available to the user device 710, which may include processes and / or applications associated with server 730 for receiving output data anomaly reports.

[0064] The user device 710, the data vendor server 745, and the server 730 may each include one or more processors, memory, and other suitable components for executing instructions, such as program code and / or data, stored on one or more computer-readable media to implement the various applications, data, and steps described herein. For example, such instructions may be stored on one or more computer-readable media, such as memory or data storage devices, that are internal to and / or external to the various components of the system 700 and / or accessible via the network 760.

[0065] The user device 710 may be implemented as a communication device that can utilize appropriate hardware and software configured for wired and / or wireless communication with the data vendor server 745 and / or server 730. For example, in one embodiment, the user device 710 may be implemented as other types of computing devices that can transmit and / or receive data, such as an autonomous vehicle, a personal computer (PC), a smartphone, a laptop / tablet computer, a wristwatch with appropriate computer hardware resources, eyeglasses with appropriate computer hardware (e.g., GOOGLE GLASS®), other types of wearable computing devices, an embedded communication device, and / or an iPad® from APPLE®. Although only one communication device is shown, multiple communication devices may function similarly.

[0066] The user device 710 in Figure 7 includes a user interface (UI) application 712 and / or other applications 716, which may correspond to executable processes, procedures, and / or applications with associated hardware. For example, the user device 710 may receive a message representing code or text output from the server 730 and display the message via the UI application 712. In other embodiments, the user device 710 may include additional or different modules having dedicated hardware and / or software, as needed.

[0067] In various embodiments, the user device 710 includes other applications 716, which may be desired in certain embodiments to provide functionality to the user device 710. For example, other applications 716 may include security applications for implementing client-side security functions, program client applications for interfacing with appropriate application programming interfaces (APIs) via the network 760, or other types of applications. Other applications 716 may include communication applications such as email, texting, voice, social networking, and IM applications that enable the user to send and receive emails, phone calls, texts, and other notifications via the network 760. For example, other applications 716 may be email or instant messaging applications that receive predictive result messages from the server 730. Other applications 716 may include device interfaces and other display modules that can receive input and / or output information. For example, other applications 716 may include processor-executable asset management software programs, including a graphical user interface (GUI) configured to provide the user 740 with an interface for viewing code or text output generated according to a particular code-related task.

[0068] The user device 710 may further include a database 718 stored in the temporary and / or non-temporary memory of the user device 710, which stores various applications and data and can be used while various modules of the user device 710 are running. The database 718 may store user profiles associated with user 740, predictions previously viewed or saved by user 740, historical data received from server 730, and the like. In some embodiments, the database 718 may be local to the user device 710. However, in other embodiments, the database 718 may be external to the user device 710 and accessible by the user device 710, which includes a cloud storage system and / or database accessible via network 760.

[0069] The user device 710 includes at least one network interface component 717 adapted to communicate with the data vendor server 745 and / or server 730. In various embodiments, the network interface component 717 may include a DSL (e.g., digital subscriber line) modem, a PSTN (public switched telephone network) modem, an Ethernet® device, a broadband device, a satellite device, and / or various other types of wired and / or wireless network communication devices, including microwave, radio frequency, infrared, Bluetooth®, and short-range wireless communication devices.

[0070] The data vendor server 745 may correspond to a server that hosts a database 719 and provides a training dataset to server 730 containing unimodal code-only data 102 and bimodal code-text pairs 104. The database 719 may be implemented by one or more relational databases, distributed databases, cloud databases, etc.

[0071] The data vendor server 745 includes at least one network interface component 726 adapted to communicate with user devices 710 and / or server 730. In various embodiments, the network interface component 726 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., digital subscriber line) modems, PSTN (public switched telephone network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency, infrared, Bluetooth®, and short-range wireless communication devices. For example, in one implementation, the data vendor server 745 may transmit asset information from the database 719 to server 730 via the network interface 726.

[0072] Server 730 may house the code understanding and generation module 630 and its submodules as shown in Figure 6. In some implementations, the code understanding and generation module 630 may receive data from the database 719 at the data vendor server 745 via the network 760 to generate code snippets or text descriptions. The generated code snippets or text descriptions may be sent via the network 760 to a user device 710 for review by the user 740.

[0073] The database 732 may be stored in the temporary and / or non-temporary memory of the server 730. In one implementation, the database 732 may store data obtained from the data vendor server 745. In one implementation, the database 732 may store parameters of the code understanding and generation module 630. In one implementation, the database 732 may store previously generated code snippets or text descriptions and their corresponding input feature vectors.

[0074] In some embodiments, the database 732 may be local to the server 730. However, in other embodiments, the database 732 may be external to the server 730 and accessible by the server 730, which includes a cloud storage system and / or database accessible via the network 760.

[0075] Server 730 includes at least one network interface component 733 adapted to communicate with user devices 710 and / or data vendor servers 745, 770, or 780 via network 760. In various embodiments, the network interface component 733 may include various other types of wired and / or wireless network communication devices, including DSL (e.g., digital subscriber line) modems, PSTN (public switched telephone network) modems, Ethernet® devices, broadband devices, satellite devices, and / or microwave, radio frequency (RF), and infrared (IR) communication devices.

[0076] Network 760 can be implemented as a single network or a combination of multiple networks. For example, in various embodiments, network 760 may include the Internet or one or more intranets, a fixed telephone network, a wireless network, and / or other suitable types of networks. Thus, network 760 can correspond to small-scale communication networks, such as private or local area networks, or large-scale networks, such as wide area networks or the Internet, accessible by various components of system 700. Exemplary Workflow

[0077] Figure 8 is an exemplary logic flow diagram illustrating how to train an encoder-decoder-based framework for code-related tasks based on the frameworks shown in Figures 1 to 7, according to some embodiments described herein. One or more of the processes of Method 800 may be implemented in the form of executable code stored on a non-temporary, tangible, machine-readable medium, which, when executed by one or more processors, can cause one or more processors to execute one or more of the processes. In some embodiments, Method 800 corresponds to the operation of a code understanding and generation module 630 (e.g., Figures 6 to 7) that performs training of an encoder-decoder-based framework for code-related tasks.

[0078] As illustrated, Method 800 includes several enumerated steps, but embodiments of Method 800 may include additional steps before, after, and between the enumerated steps. In some embodiments, one or more of the enumerated steps may be omitted or performed in a different order.

[0079] In step 801, a first training dataset of unimodal code data and a second training dataset of bimodal code-text pair data may be received via a communication interface (e.g., data interface 615 in Figure 6, network interface 733 in Figure 7).

[0080] In step 803, the encoder (e.g., 110a in Figure 3A) may encode the first training input into a first coded representation based on a first code sequence from the first training dataset. For example, the first training input is generated by randomly replacing some of the tokens in the first code sequence with indexed sentinel tokens. In another example, the first training input is generated by randomly selecting a pivot position in the first code sequence and then cutting out a first portion of the first code sequence prior to the pivot position as the first training input.

[0081] In step 805, the decoder (e.g., 110b in Figure 3A) can generate a code output from the first code representation.

[0082] In step 807, at least one unimodal training objective is calculated based on the code output and the first code sequence. For example, at least one unimodal training objective is calculated by comparing the code sequence reconstructed by the decoder with the first code sequence. In another example, at least one unimodal training objective is calculated by comparing the remaining portion of the code predicted by the decoder with a second portion of the first code sequence after the pivot position.

[0083] In step 809, the encoder and decoder may be trained together according to at least one unimodal training objective.

[0084] In step 811, the pre-trained encoder can encode text from the second training input and the second training dataset of the second code sequence into a second code representation and a second text representation. For example, the encoder comprises a first encoder module and a second encoder module that share the same parameters and operate in parallel. The first encoder module encodes text into a text representation, and the second encoder module encodes the second code sequence into a second code representation in parallel.

[0085] In step 813, the pre-trained decoder may generate training outputs from the second coded and textual representations. For example, the decoder comprises one or more decoder modules that share the same parameters except for the last feedforward layer, which is adapted as a different decoder head for different bimodal training objectives. One or more decoder modules may generate their respective decoded outputs in parallel according to different bimodal training objectives.

[0086] In step 815, at least one bimodal training objective may be calculated based on the training output and the second training input. For example, at least one bimodal training objective is calculated as a control loss based on a batch of code-text pairs from the second training dataset. The encoder generates a set of code representations and a set of text representations from the batch of code-text pairs. A set of code-text similarity and a set of text-code similarity are calculated between the set of code representations and the set of text representations (e.g., equations (1)-(2)). The control loss is calculated by comparing the set of code-text similarity and the set of text-code similarity with the ground truth one-hot similarity (e.g., equation (3)).

[0087] In another example, at least one bimodal training objective is calculated as a text-code matching loss. The decoder's classification head may generate predictions about whether a second code sequence and text are matching pairs. The matching loss is calculated by comparing the predictions with the ground truth one-hot labels corresponding to the second code sequence and text (e.g., equation (4)).

[0088] In another example, at least one bimodal training objective is calculated as text-to-code generation loss. The decoder can generate a predicted code sequence based on the text representation. The text-to-code generation loss is calculated by comparing the predicted code sequence with a second code sequence.

[0089] In another example, at least one bimodal training objective is calculated as the code-text generation loss. The decoder can generate predicted text based on the second coded representation. The code-text generation loss is calculated by comparing the predicted text with the actual text.

[0090] In step 817, the pre-trained encoder and pre-trained decoder are (again) trained according to at least one bimodal training objective. For example, the pre-trained encoder and pre-trained decoder may be trained according to a sum of multiple bimodal training objectives, e.g., equation (5).

[0091] In step 819, the trained encoder and decoder models can perform code-related tasks such as code summarization, code generation, code retrieval, and code-text alignment. For example, depending on the type of code-related task, an encoder-only model, a decoder-only model, or an encoder-decoder model may be selected to perform the code-related task. Illustrative data experiment

[0092] Figures 9 to 14 provide exemplary data tables showing the results of data experiments on the encoder-decoder pre-training framework described in Figures 1 to 8, according to embodiments described herein.

[0093] Comprehensive experiments were conducted across nine programming languages ​​(PLs) for a wide range of code comprehension and generation tasks. Due to gaps in the task or data domain with the second stage of bimodal pre-training, checkpoints from the first stage of pre-training are used for code completion and mathematical programming tasks. Two variations of encoder-decoder model 110, namely the base(220M) model and the large(770M) model, are employed. Encoder-decoder model 110 (referred to as "CodeT5Mix" in the data experiments) is compared to a pre-trained LM on a codebase.For encoder-only models, there are RoBERTa (Liu et al., A robustly optimized BERT pretraining approach, in Computing Research Repository (CoRR), abs / 1907.11692, 2019), CodeBERT trained with mask language modeling (Feng et al., Codebert: A pre-trained model for programming and natural languages, In proceedings of Empirical Methods in Natural Language Processing (EMNLP), pp. 1536-1547, 2020), GraphCodeBERT which uses data flow extracted from the abstract syntax tree (AST) of the code (Guo et al., GraphCodeBERT: Pre-training code representations with data flow, In proceedings of International Conference and Learning Representations (ICLR), OpenReview.net, 2021), and SYNCOBERT which incorporates AST and contrastive learning (Wang et al., Syncobert: Syntax-guided multi-modal contrastive pre-training for code The representation (arXiv preprint arXiv:2108.04556, 2021) is taken into consideration.For decoder-only models, GPT-2 (Radford et al., Language models are unsupervised multitask learners, OpenAI blog, 1(8):9, 2019) and CodeGPT (Lu et al., Codexglue: A machine learning benchmark dataset for code understanding and generation, In proceedings of NeurIPS Datasets and Benchmarks, 2021) are considered, both of which are pre-trained using the CLM objective. For encoder-decoder models, PLBART (Ahmad et al., Unified pre-training for program understanding and generation, In proceedings of North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 2655-2668, 2021) and CodeT5 (US Patent Nonprovisional Application No. 17 / 459,968), which uses an integrated framework to support both understanding and generation tasks, are considered. The CodeT5-large results for U.S. Patent Nonprovisional Application No. 17 / 896,942, filed on August 26, 2022, which is concurrently pending and jointly owned, will also be compared.

[0094] In addition, another unified model, UniXcoder (Guo et al., Unixcoder: Unified cross-modal pre-training for code representation, In proceedings of Association for Computational Linguistics (ACL), pp. 7212-7225, 2022), which uses UniLM-style masking (Dong et al., Unified language model pre-training for natural language understanding and generation, In proceedings of NeurIPS 2019, pp. 13042-13054, 2019), is considered. Models such as CodeBERT, GraphCodeBERT, SYNCOBERT, and UniXcoder are based on RoBERTa-base with 125M parameters, GPT-2 / CodeGPT has 124M, and PLBART has 140M. In particular, CodeT5Mix uses only half of its full size when operating in encoder-only mode and decoder-only mode.

[0095] The text-to-code search task aims to find the most semantically relevant code snippet at the functional level from a set of candidate codes based on a natural language query. Three datasets were used for evaluation: CodeSearchNet (Husain et al., CodeSearchNet challenge: Evaluating the state of semantic code search, CoRR, abs / 1909.09436, 2019), CosQA (Huang et al., CosQA: 20,000+ web queries for code search and question answering. In ACL / IJCNLP (1), pp. 5690-5700, Association for Computational Linguistics, 2021), and AdvTest (Lu et al.). These were curated from the original CodeSearchNet by filtering the data with low-quality queries, incorporating real-world queries from modern search engines, and normalizing the code by obfuscating identifiers. In this task, the bimodal encoder and matching decoder of CodeT5Mix are activated, and the mean inverse rank (MRR) is used as the metric.

[0096] As shown in Table 1 of Figure 9, CodeT5Mix-base performs significantly better than all existing encoder-only and encoder-decoder models, and the modified large further sets new SoTA results, outperforming the previous SoTA UniXcoder by more than 3 absolute MRR points on all three tasks across eight datasets. This means that CodeT5Mix is ​​a robust code retriever model for handling queries with diverse formats and PLs. Furthermore, CodeT5Mixbase shows a significant performance improvement over CodeT5-base, which may be due to text-code contrast learning and matching objectives that facilitate better unimodal and bimodal representation learning. In particular, compared to SYNCOBERT and UniXcoder pre-trained in contrast learning, the proposed model achieves much better results, which may be due to the bimodal matching decoder that enables finer-grained text-code alignment.

[0097] CodeT5Mix's decoder-only generation capability is tested through line-level code completion tasks, which aim to complete the next line of code based on the previous code context. The PY150 from CodeXGLUE (Raychev et al., Probabilistic model for code with decision trees. In proceedings of Object-Oriented Programming Systems, Languages, and Applications (OOPSLA), pp. 731-747, 2016) and GitHub JavaCorpus (Allamanis et al., Mining source code repositories at massive scale using language modeling, in proceedings of Mining Software Repositories (MSR), pp. 207-216, IEEE Computer Society, 2013) use perfect match (EM) accuracy and Levenshtein edit similarity (Svyatkovskiy et al., Intellicode compose: code generation using transformer, In proceedings of ESEC / SIGSOFT FSE, pp. 1433-1443, 2020) as evaluation metrics. Typically, this task requires a decoder-only model for efficient training.

[0098] As shown in Table 3 of Figure 11, CodeT5Mix achieves new state-of-the-art results compared to both decoder-only and encoder-decoder models across both metrics. In particular, CodeT5Mix-base shows a significant improvement over CodeT5-base, with EM scores of 6.45 and 9.43 in PY150 and JavaCorpus, respectively. This is primarily due to the CLM objective of the present invention in the first stage of pre-training, which allows the decoder to see longer sequences rather than discrete span combinations in CodeT5, leading to improved causal generation capabilities.

[0099] The code summarization task aims to summarize code snippets into docstrings, while code generation aims to generate functions based on natural language descriptions. Clean versions of the CodeSearchNet dataset across six PLs are used for code summarization, and Java ConCode (Iyer et al., Mapping language to code in programmatic context, In proceedings of EMNLP, pp. 1643-1652, 2018) is used for code generation. For evaluation metrics, BLEU-4 (B4), exact match (EM) precision, and CodeBLEU (CB) (Ren et al., CodeBLEU: a method for automatic evaluation of code synthesis, CoRR, abs / 2009.10297, 2020), which considers syntactic and semantic matching based on code structure in addition to n-gram matching, are employed.

[0100] In CodeT5Mix, the bimodal encoder and text generation decoder are activated for code summarization, and the code generation decoder is activated for code generation. From Table 2 in Figure 10, the encoder-decoder models (CodeT5 and CodeT5Mix) generally perform better than both encoder-only and decoder-only models, as well as the integrated UniXcoder with controlled masks for both tasks. This means that the encoder-decoder models can be better supported.

[0101] For the seq2Seq generation task, CodeT5Mix-large achieves new state-of-the-art results for both tasks across various metrics. To evaluate the model for code generation, exact match or BLEU score may be limited, as multiple forms of the exact program solution may exist. Two mathematical programming tasks, namely MathQA-Python (Austin et al., Program synthesis with large language models, arXiv preprint arXiv:2108.07732, 2021) and GSM8K (Cobbe et al., Training verifiers to solve math word problems, CoRR, abs / 2110.14168, 2021), allow for measuring the correctness of the code based on the execution output of the code program. The task is to generate Python programs to solve mathematical problems described in natural language. Solutions in GSM8K are converted into Python programs (hereinafter, GSM8K-Python), an example of which is shown in Figure 13.

[0102] Pass@k is used, which measures the percentage of problems solved using k generated programs for each problem. In addition to CodeT5, CodeT5Mix utilizes self-sampling optimization (Ni et al., Learning from self-sampled correct and partially-correct programs, CoRR, abs / 2205.14318, 2022) for Codex (Chen et al., Evaluating large language models trained on code, arXiv preprint arXiv:2107.03374, 2021), LaMDA (Austin et al., Program synthesis with large language models, arXiv preprint arXiv:2108.07732, 2021), PaLM-Coder (Chowdhery et al., PALM: Scaling language modeling with pathways, CoRR, abs / 2204.02311, 2022), and GPT-Neo (Black et al., Gpt-NEO: Large scale autoregressive language modeling with mesh-tensorflow, March This is compared to a very large decoder-only model, including (2021, URL https: / / doi.org / 10.5281 / zenodo, 5297715). As shown in Table 4 of Figure 13, CodeT5Mix achieves a significant performance improvement, outperforming many pre-trained models of much larger size. Specifically, CodeT5Mix-large achieves new state-of-the-art results of 87.4 passes@80 in MathQA-Python and 73.8 passes@100 in GSM8K-Python. In GSM8K-Python, the encoder-decoder model achieves the second-best result of 26.2 passes@1, second only to PaLM-Coder, which has a much larger size (540B) than CodeT5Mix.This model was also exposed to a much larger amount of pre-training data, which provided additional explanations for the solutions.

[0103] As shown in Figure 14, CodeT5Mix achieves significantly better results in all categories, particularly in the retrieval-based and RA generation settings shown in Figure 5. Specifically, it performs two code generation tasks by reversing the order of input and output for code summarization on Java® and Python and using their publicly available, deduplicated retrieval codebases. CodeT5Mix can be evaluated in three settings: retrieval-based, generative, and retrieval-augmented generative. In the retrieval-based setting, a bimodal encoder is activated to search for the top 1 code sample as a prediction given a text query, while in the RA generation setting, the top k combinations of retrieved samples (k=1 in this experiment) are appended to the input to activate the code generation decoder.

[0104] The previous state-of-the-art model, REDCODER-EXT (Parvez et al., Retrieval augmented code generation and summarization, In proceedings of EMNLP (Findings), pp. 2719-2734, 2021), uses GraphCodeBERT as the retriever and PLBART as the generator separately, but the CodeT5Mix model can be flexibly used as an end-to-end system that combines both retrieval and generation capabilities.

[0105] This description and accompanying drawings illustrating aspects, embodiments, implementations, or applications of the present invention should not be construed as limiting. Various mechanical, compositional, structural, electrical, and operational modifications can be made without departing from the spirit and scope of this specification and the claims. In some cases, well-known circuits, structures, or techniques are not shown or described in detail so as not to obscure the embodiments of this disclosure. Similar numbers in two or more figures represent the same or similar elements.

[0106] This description provides specific details illustrating several embodiments consistent with the present disclosure. Numerous specific details are provided to provide a complete understanding of the embodiments. However, it will be apparent to those skilled in the art that some embodiments can be implemented without some or all of these specific details. The specific embodiments disclosed herein are illustrative and not limiting. Those skilled in the art can implement other elements not specifically described herein but within the scope and spirit of the disclosure. Furthermore, to avoid unnecessary repetition, one or more features shown and described in relation to one embodiment may be incorporated into other embodiments unless otherwise noted, or if one or more features would render the embodiment non-functional.

[0107] While exemplary embodiments have been shown and described, extensive modifications, alterations, and substitutions are intended in the foregoing disclosure, and in some cases, some features of the embodiments may be used without corresponding use of other features. Those skilled in the art will recognize many variations, substitutions, and modifications. Therefore, the scope of the present invention should be limited only by the following claims, which should be interpreted to broadly coincide with the scope of the embodiments disclosed herein.

Claims

1. A method for training an encoder-decoder-based framework for code-related tasks, The steps include receiving a first training dataset of unimodal code data via a communication interface, The steps include: encoding the first training input into a first coded representation based on a first code sequence from the first training dataset using an encoder; The decoder generates a code output from the first code representation, A step of calculating at least one unimodal training objective based on the code output and the first code sequence, A step of pre-training the encoder and the decoder in accordance with at least one unimodal training objective, The steps include receiving a second training dataset of bimodal code-text pair data via a communication interface, The steps include: encoding the text from the second training input of the second code sequence and the second training dataset into a second code representation and a second text representation using the pre-trained encoder; The steps include generating a training output from the second code representation and the text representation using the pre-trained decoder, A step of calculating at least one bimodal training objective based on the training output and the second training input, A step of training the pre-trained encoder and the pre-trained decoder in accordance with at least one bimodal training objective: Methods that include...

2. The first training input is generated by randomly replacing some of the tokens in the first code sequence with indexed sentinel tokens. The at least one unimodal training objective is calculated by comparing the code sequence reconstructed by the decoder with the first code sequence. The method according to claim 1.

3. The first training input is, The steps include randomly selecting a pivot position within the first code sequence, A step of cutting out a first portion of the first code sequence prior to the pivot position as the first training input. Generated by, The at least one unimodal training objective is calculated by comparing the predicted remainder of the code by the decoder with a second portion of the first code sequence after the pivot position. The method according to claim 1.

4. The at least one bimodal training objective is based on a batch of code-text pairs from the second training dataset, The encoder generates a set of code representations and a set of text representations from a batch of code-text pairs. The steps include calculating a set of code-text similarity and a set of text-code similarity between the set of code representations and the set of text representations, The steps include: calculating a control loss by comparing the set of code-text similarity values ​​and the set of text-code similarity values ​​with ground truth one-hot similarity values; The method according to claim 1, calculated by...

5. The aforementioned at least one bimodal training objective is The steps include: generating a prediction by the classification head of the decoder regarding whether the second code sequence and the text are a matching pair; The steps include: calculating the matching loss by comparing the Ground Truth One hot label corresponding to the second code sequence and the text with the prediction; The method according to claim 1, calculated by...

6. The aforementioned at least one bimodal training objective is The steps include: generating a predictive code sequence based on the text representation using the decoder; A step of calculating text-code loss by comparing the predicted code sequence with the second code sequence. The method according to claim 1, calculated by...

7. The aforementioned at least one bimodal training objective is The decoder generates predictive text based on the second code representation, A step of calculating code-text loss by comparing the predicted text with the text. The method according to claim 1, calculated by...

8. The steps include training the pre-trained encoder and the pre-trained decoder together according to the sum of multiple bimodal training objectives, and The method according to claim 1, further comprising:

9. The encoder comprises a first encoder module and a second encoder module that share the same parameters and operate in parallel, and the step of encoding the second training input is: The first encoder module encodes the text into the text representation, The second encoder module encodes the second code sequence in parallel with the second code representation. The method according to claim 1, including the method described in claim 1.

10. The decoder comprises one or more decoder modules that share the same parameters except for the last feedforward layer, which is adapted as different decoder heads for different bimodal training purposes, and the step of generating the training output is: The step of generating separate decoded outputs in parallel according to the different bimodal training objectives using one or more decoder modules. The method according to claim 1, including the method described in claim 1.

11. The step of generating a code-related task output in response to a code-related task input according to a specific code-related task using only the trained encoder. The method according to claim 1, further comprising:

12. The step of using only the trained decoder to generate a code-related task output in response to a code-related task input, according to a specific code-related task. The method according to claim 1, further comprising:

13. The steps include: encoding the code-related task input into a task representation using the trained encoder; The steps include: using the trained decoder to generate a code-related task output from the task representation according to a specific code-related task; The method according to claim 1, further comprising:

14. A system for training encoder-decoder-based frameworks for code-related tasks, An encoder, a decoder, and memory for storing multiple processor-executable instructions, A communication interface that receives a first training dataset of unimodal code data and a second training dataset of bimodal code-text pair data, One or more hardware processors read and execute the plurality of processor-executable instructions from the memory in order to perform an operation. The operation is provided, The steps include: encoding the first training input into a first coded representation based on a first code sequence from the first training dataset using the encoder; The decoder generates a code output from the first code representation, A step of calculating at least one unimodal training objective based on the code output and the first code sequence, A step of pre-training the encoder and the decoder in accordance with at least one unimodal training objective, The steps include: encoding the text from the second training input of the second code sequence and the second training dataset into a second code representation and a second text representation using the pre-trained encoder; The steps include generating a training output from the second code representation and the text representation using the pre-trained decoder, A step of calculating at least one bimodal training objective based on the training output and the second training input, A step of training the pre-trained encoder and the pre-trained decoder in accordance with at least one bimodal training objective: A system that includes this.

15. The encoder comprises a first encoder module and a second encoder module that share the same parameters and operate in parallel, and the step of encoding the second training input is: The first encoder module encodes the text into the text representation, The second encoder module encodes the second code sequence in parallel with the second code representation. The system according to claim 14, including the system described in claim 14.

16. The decoder comprises one or more decoder modules that share the same parameters except for the last feedforward layer, which is adapted as different decoder heads for different bimodal training purposes, and the step of generating the training output is: The step of generating separate decoded outputs in parallel according to the different bimodal training objectives using one or more decoder modules. The system according to claim 14, including the system described in claim 14.

17. A non-temporary machine-readable medium containing a plurality of machine-executable instructions, wherein, when executed by one or more processors, the plurality of machine-executable instructions are adapted to cause the one or more processors to perform an operation, and the operation is The steps include receiving a first training dataset of unimodal code data via a communication interface, The steps include: encoding the first training input into a first coded representation based on a first code sequence from the first training dataset using an encoder; The decoder generates a code output from the first code representation, A step of calculating at least one unimodal training objective based on the code output and the first code sequence. A step of pre-training the encoder and the decoder in accordance with at least one unimodal training objective, The steps include receiving a second training dataset of bimodal code-text pair data via a communication interface, The steps include: encoding the text from the second training input of the second code sequence and the second training dataset into a second code representation and a second text representation using the pre-trained encoder; The steps include generating a training output from the second code representation and the text representation using the pre-trained decoder, A step of calculating at least one bimodal training objective based on the training output and the second training input, A step of training the pre-trained encoder and the pre-trained decoder in accordance with at least one bimodal training objective: Non-temporary machine-readable media, including [specific examples of such media].

18. The encoder comprises a first encoder module and a second encoder module that share the same parameters and operate in parallel, and the step of encoding the second training input is: The first encoder module encodes the text into the text representation, The second encoder module encodes the second code sequence in parallel with the second code representation. A non-temporary machine-readable medium according to claim 17, including the following:

19. The decoder comprises one or more decoder modules that share the same parameters except for the last feedforward layer, which is adapted as different decoder heads for different bimodal training purposes, and the step of generating the training output is: The step of generating separate decoded outputs in parallel according to the different bimodal training objectives using one or more decoder modules. A non-temporary machine-readable medium according to claim 17, including the following:

20. The aforementioned operation is, The step of generating a code-related task output in response to a code-related task input according to a specific code-related task, using only the trained encoder, only the trained decoder, or both the trained encoder and decoder. A non-temporary machine-readable medium according to claim 17, further comprising:

Images