A Neural Network-Based Intelligent Programming Language Program Translation Method System

Abstract

This invention proposes a neural network-based intelligent programming language program translation method and system, comprising: constructing a forward model for translating a source language program into a target language program, and a reverse model for translating a target language program into a source language program; training the forward model and the reverse model through reverse translation based on a source language program library and a target language program library to obtain a forward translation model and a reverse translation model; inputting the source language program to be translated into the forward translation model, inputting multiple candidate results into a reordering model to obtain scores for each candidate result, and selecting the candidate result with the highest score as the translation result of the source language program.

Description

Technical Field

[0001] This invention relates to the fields of program code conversion, program generation, program translation, and natural language processing, and particularly to a method system for translating intelligent programming language programs based on neural networks. Background Technology

[0002] Intelligent chips, as a crucial physical carrier of deep learning algorithms, have been widely used in various fields. As a vital component of system software in intelligent computing systems, dedicated intelligent programming languages, such as CUDA and BANG, are constantly emerging. Due to the rapid development of deep learning algorithms, developers need to develop user programs using these intelligent programming languages ​​to deploy custom algorithms on intelligent chips. However, these languages ​​are highly complex and demanding on programmers. Because of the large computing units in intelligent chips, which often exhibit parallelism in data and tasks, programmers must adhere to parallel programming models when using these languages. Furthermore, unlike the transparent storage structures such as caches on general-purpose processors, programmers using intelligent programming languages ​​must manually declare and manage different types of storage spaces. These challenges undoubtedly increase the difficulty for programmers developing user programs on intelligent chips.

[0003] User programs for traditional programming languages ​​based on the serial programming model are readily available, thanks to two main reasons: (1) a large amount of legacy code that has already been developed and optimized exists on the serial programming model; and (2) the serial programming model conforms to the usage habits of programmers, allowing users to easily write user programs using traditional programming languages. Therefore, if user programs from traditional programming languages ​​can be migrated to user programs from intelligent programming languages, the overhead of programmers developing user programs on intelligent chips can be greatly reduced.

[0004] Currently, some works have been used for translation between different natural languages ​​(such as English to Chinese), and others for programming language translation (such as translating C to CUDA). The existing technology TransCoder can use neural networks for programming language translation, and it can automatically translate user programs such as C, Python, and Java. It consists of a basic Transformer architecture, containing an encoder and a decoder, each with six independent layers. This model is shared across multiple programming languages ​​and is trained using three unsupervised training methods.

[0005] The first task is cross-programming language pre-training, which is similar to the training method of Masked Language Model (MLM). A portion of the words in the input sequence is randomly masked, and the model is trained to predict the words based on the context of the masked words. During training, training data in different languages ​​are alternated to allow the model to learn representations of different languages ​​with higher quality.

[0006] The second task: Denoising Autoencoder: After pre-training with the masked language model, the encoder has learned word representations. However, translation tasks require the joint participation of both the encoder and decoder, and at this point, the decoder has not yet learned how to decode. Therefore, TransCoder introduces a denoising autoencoder task to train the decoder. This method involves adding noise to the input sequence, such as random occlusion, deleting a word, or shuffling its order, and then having the model predict the original sequence. This training method involves the decoder in the training process, and also makes the model more stable and able to handle input noise.

[0007] The third task: reverse translation. After the first two training steps, the model can generate preliminary translation results. However, the quality of the generated results is poor because the model cannot yet translate from one specified language to another. To solve this problem, reverse translation is introduced. Reverse translation is a solution for situations with unsupervised corpora; it is a weakly supervised learning model. Reverse translation consists of two parts: a source language to target language translation model and a target language to source language translation model. First, the target language corpus is used as input, and the model translates it to obtain the source language. This generates a noisy source language corpus. This source language corpus and the target language corpus can then be approximated as a pair of supervised corpora. Therefore, the process from source language to target language can be carried out in a weakly supervised manner. During training, both directions are trained simultaneously until convergence.

[0008] The following mainly discusses the problems and shortcomings of existing Transcoder technologies.

[0009] Transcoders are primarily designed for traditional serial programming languages ​​such as C, Java, and Python. Translating between these languages, which are semantically similar (meaning they are all serial programs) and differ only in syntax (referring to different keywords in different programming languages, such as `void` in C and `def` in Python), is relatively easy. However, traditional programming languages, such as C (hereafter referred to as C as a representative of traditional programming languages), are serial programming models that describe complex operations using scalar representations with nested loop structures. Intelligent programming languages, such as CUDA (hereafter referred to as CUDA as a representative of intelligent programming languages), are parallel programming models that require parallel data processing. The C programming model and the CUDA programming model differ significantly in syntax and semantics. At the syntactic level, the CUDA programming model has special keywords, such as the parallel variable `threadIdx.x`. At the semantic level, the nested loop structures in C programs need to be translated into parallel variables in CUDA programs. These issues significantly increase the difficulty of automatically translating C programs into CUDA programs.

[0010] Existing transcoder technology does not analyze program semantics, especially parallel semantics. Therefore, it performs poorly when faced with the problem of translating C language into CUDA language, and cannot effectively detect loop structures in C serial programs and translate them into corresponding CUDA programs. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing technologies that fail to consider the semantic differences between intelligent programming languages ​​and traditional programming languages. This invention proposes an index for modeling parallel semantics and a method and system for reordering generated programs. The shortcomings of existing technologies can be overcome by introducing modeling of parallel semantics and using an additional reordering model to select the optimal solution from the generated CUDA program.

[0012] Existing technology, Transcoder, can be considered as a translation model that takes a C program as input and outputs a set of CUDA programs as candidates. Then, using the novel reordering model introduced in this invention, these programs are scored based on parallel semantic modeling metrics to select the optimal CUDA program.

[0013] Specifically, this invention proposes a neural network-based intelligent programming language program translation method, which includes:

[0014] Step 1: Construct a forward model for translating a source language program into a target language program, and a reverse model for translating a target language program into a source language program; based on the source language program library and the target language program library, train the forward model and the reverse model through reverse translation to obtain the forward translation model and the reverse translation model;

[0015] Step 2: Input the source language program to be translated into the forward translation model, input the multiple candidate results into the reordering model, obtain the score of each candidate result, and select the candidate result with the highest score as the translation result of the source language program.

[0016] The aforementioned neural network-based intelligent programming language program translation method, wherein step 1 includes:

[0017] Select a program from the source language program library as the first training target and input it into the forward model to obtain the first target language. Input the intermediate target language into the reverse model to obtain the first source language. Construct the first loss based on the first source language and the first training target, and train the forward model and the reverse model.

[0018] Simultaneously, a program is selected from the target language program library as the second training target and input into the inverse model to obtain the second source language. The second source language is then input into the forward model to obtain the second target language. A second loss is constructed based on the second target language and the second training target, and the forward model and the inverse model are trained.

[0019] Until the first loss and the second loss converge, the current forward model and the backward model are saved as the forward translation model and the backward translation model, respectively.

[0020] The aforementioned neural network-based intelligent programming language program translation method, wherein the training of the reordering model includes:

[0021] Select program x from the target language program library and input it into the back translation model to generate an intermediate source program. Input the intermediate source program into the forward translation model to obtain multiple candidate results. Normalize the prediction score of the reordering model for each candidate result and obtain the output distribution.

[0022] Using the program x as a baseline, the correct score of each candidate result is obtained as the output distribution. The re-ranking model is trained by minimizing the divergence between the output distribution and the output distribution.

[0023] The aforementioned neural network-based intelligent programming language program translation method, wherein the source language is C, Python, or Java, and the target language is CUDA or BANG.

[0024] This invention also proposes an intelligent programming language program translation system based on neural networks, which includes:

[0025] The initial module is used to construct a forward model that translates a source language program into a target language program, and a reverse model that translates a target language program into a source language program. Based on the source language program library and the target language program library, the forward model and the reverse model are trained by reverse translation to obtain the forward translation model and the reverse translation model.

[0026] The translation module is used to input the source language program to be translated into the forward translation model, input the multiple candidate results into the reordering model, obtain the score of each candidate result, and select the candidate result with the highest score as the translation result of the source language program.

[0027] The aforementioned neural network-based intelligent programming language program translation system, wherein the initial module is used for:

[0028] Select a program from the source language program library as the first training target and input it into the forward model to obtain the first target language. Input the intermediate target language into the reverse model to obtain the first source language. Construct the first loss based on the first source language and the first training target, and train the forward model and the reverse model.

[0029] Simultaneously, a program is selected from the target language program library as the second training target and input into the inverse model to obtain the second source language. The second source language is then input into the forward model to obtain the second target language. A second loss is constructed based on the second target language and the second training target, and the forward model and the inverse model are trained.

[0030] Until the first loss and the second loss converge, the current forward model and the backward model are saved as the forward translation model and the backward translation model, respectively.

[0031] The aforementioned neural network-based intelligent programming language program translation system, wherein the training of the reordering model includes:

[0032] Select program x from the target language program library and input it into the back translation model to generate an intermediate source program. Input the intermediate source program into the forward translation model to obtain multiple candidate results. Normalize the prediction score of the reordering model for each candidate result and obtain the output distribution.

[0033] Using the program x as a baseline, the correct score of each candidate result is obtained as the output distribution. The re-ranking model is trained by minimizing the divergence between the output distribution and the output distribution.

[0034] The aforementioned neural network-based intelligent programming language program translation system, wherein the source language is C, Python, or Java, and the target language is CUDA or BANG.

[0035] The present invention also proposes a storage medium for storing a program that executes any of the neural network-based intelligent programming language program translation methods.

[0036] The present invention also proposes a client for any of the aforementioned neural network-based intelligent programming language program translation systems.

[0037] As can be seen from the above solutions, the advantages of the present invention are:

[0038] This invention enables the automatic translation of programs from traditional programming languages ​​to intelligent programming languages, achieving a BLEU (bilingual evaluation understudy) score improvement from 72.21 to 74.00 (an increase of 1.79) compared to existing technologies. The compilation success rate of the generated intelligent programming language programs increases from 83.8% to 92.8%. Furthermore, the generated intelligent programming language programs can achieve up to 347 times faster execution speed compared to the original traditional programming language programs. This invention can also assist programmers in writing user programs, increasing their development efficiency in intelligent programming language programs by up to 3.8 times. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating the overall training process of the present invention.

[0040] Figure 2 This is a flowchart of the first and second phases of training in this invention;

[0041] Figure 3 This is a flowchart of the third stage training process of the present invention;

[0042] Figure 4 This is a flowchart of the fourth stage training process of the present invention;

[0043] Figure 5 This is a flowchart illustrating the overall translation process of this invention. Detailed Implementation

[0044] To achieve the above-mentioned technical effects, the present invention mainly includes the following key technical points:

[0045] Key point 1, parallel semantic modeling; technical effect: able to score programs written in intelligent programming languages ​​and evaluate the parallel semantics of the generated programs;

[0046] Key Point 2, Reordering Model (or a combination of Translation Model and Reordering Model); Technical Effect: It can utilize Key Point 1 (i.e., parallel semantic modeling) for reordering, improving the translation of programs from traditional programming languages ​​to intelligent programming languages.

[0047] To make the above features and effects of the present invention clearer and easier to understand, specific embodiments are described below, and detailed descriptions are provided in conjunction with the accompanying drawings.

[0048] Parallel semantic modeling:

[0049] This invention proposes an evaluation index, ParaBLEU, based on BLEU:

[0050] ParaBLEU=α·BLEU+β·BLEU weight +(γ·Match ast +δ·Match df )×SIM CUDAkeywords ×SIM loops ×SIM parallel

[0051] In the formula BLEU weight It is a weighted BLEU, Match represents the degree of matching, and AST is an abbreviation for Abstract Syntax Trees. ast This refers to the degree of matching between the hypothetical program and the reference program on the abstract syntax tree; df is an abbreviation for dataflow. df SIM refers to the degree of similarity between the hypothetical program and the reference program in terms of data flow. CUDA keywords are keywords related to CUDA. CUDAkeywords This assumes the similarity between the assumed program and the reference program in terms of CUDA-related keywords. Loops are loop structures and an important component of CUDA programs. SIM loops This refers to the similarity between the assumed program and the reference program in their loop structures; `parallel` expresses parallel semantics, that is, on which loop axes can be parallelized and on which loop axes cannot. SIM parallel This refers to the degree of similarity between the hypothetical program and the reference program in terms of parallel semantics. α, β, γ, and δ are preset weights, typically 0.25.

[0052] This metric takes into account keyword similarity, loop structure similarity, and parallel semantic similarity of intelligent programming languages.

[0053] Keyword matching is used to distinguish CUDA code from C code at the syntactic level, such as determining the presence of CUDA-specific keywords like `_global_`. Loop structure similarity checks the degree of matching of loop structures. Parallel semantic similarity, building on loop structure similarity, further determines whether the originally sequential program has been correctly translated into a parallel program. Specifically, this invention uses similarity distance instead of matching scores to evaluate loop structure similarity and parallel semantic similarity. Similarity distance better evaluates the similarity of parallel semantics and penalizes mismatched parallel semantics.

[0054] The ParaBLEU evaluation metric allows for better scoring of generated programs.

[0055] Reordering model (or a combination of translation model and reordering model):

[0056] A simplified flowchart of the training process:

[0057] Since C is a typical traditional programming language, it will be used as the source language for program translation in this embodiment. Since CUDA is a typical intelligent programming language, it will be used as the target language for program translation in this embodiment.

[0058] like Figure 1 As shown, the training of this invention is divided into four stages: pre-training, noise reduction autoencoder, reverse translation, and discriminative reordering.

[0059] The first stage - pre-training: such as Figure 2 As shown in the upper part, programs written in C and CUDA are collected to form C single-language corpora and CUDA single-language corpora, respectively. Then, a basic language model (LM) is obtained through pre-training. The language model enables the computer to "learn" C and CUDA, but at this stage, the relationship between C and CUDA cannot be established. Subsequent translation models will enable the computer to learn the relationship between C and CUDA.

[0060] The second stage - noise reduction auto-encoding: such as Figure 2 (See the lower half). This method trains the decoder of the translation model, mainly to make the model more stable and able to cope with input noise.

[0061] The third stage - reverse translation: such as Figure 3As shown, during the training process of reverse translation, the source language to target language conversion model C->CUDA and the target language to source language CUDA->C conversion model are trained simultaneously. Specifically, typically, due to the lack of program pairs composed of source and target languages, supervised learning methods cannot be used to train the translation model. Therefore, this invention uses a reverse translation method to generate a corresponding CUDA program for an input program in a C single-language corpus, and then translates it back into a C program. In this way, this invention can use the original input C program as a reference and the C program after C-CUDA-C as an assumption, and train the translation model (Machine Translation Model, MT Model) by minimizing the difference between the two; the same applies to the CUDA-C-CUDA direction.

[0062] The fourth stage - reordering model: such as Figure 4 As shown, after the previous back-translation training process, the translation model is capable of C-to-CUDA conversion, but the quality of the generated CUDA program is insufficient. The training process of discriminative re-ranking relies on supervised data for parallel semantic evaluation of the program. Therefore, this invention utilizes the training results of the back-translation model, as shown in the figure, to generate re-ranking model data during the CUDA-C-CUDA back-translation process.

[0063] First, given CUDA code x as input, the CUDA-C translation model generates C code u. In this process, the beam search is set to 1, meaning only a result of 1 is generated. Then, u is used as input to the C-CUDA translation model, where the beam search is set to 50, generating the result w. i ,i∈[1,50].

[0064] This invention uses these N candidate results as input to the re-ranking model D. The re-ranking model D can employ a simple MLP network. The structure is: MLP -> tanh (activation function) -> MLP. The re-ranking model predicts a ParaBLEU score for each candidate result wi, and the output distribution p of model D is obtained through normalization. D Meanwhile, this invention employs the same method, using the original CUDA input x as a baseline, and using each w i As an assumption, the target distribution p for training model D is obtained. This is achieved by minimizing p. D The Kullback-Leibler divergence between p and is used to train the reordering model D. The entire training process is shown in the formula:

[0065]

[0066] In the formula, D: reordering model; u: input to the C-CUDA translation model; w i ,i∈[1,50]. The order of 50 solutions in a beam search; Exp: exponential function; the first formula above: the distribution p of the reordering model. D The denominator is the sum of the exponents of the 50 solutions in the bundle model, and the numerator is the exponent value of each solution, representing the current solution w. i The probability. The second formula above: max min takes the maximum and minimum values, which is equivalent to a normalization function, normalizing the score to the range [0,1]. The third formula above: similar to the first formula, uses the score in formula 2 to calculate the target distribution p, where τ is a temperature parameter, which can take the value 0.5. The last formula is the fourth formula L = Loss. The left side of the formula represents the loss of the reordering model. The right side of the formula is the calculation formula for Kullback-Leibler divergence (i.e., KL divergence). KL divergence fits the distribution of the reordering model D and the target distribution calculated by the score.

[0067] After training, this invention yields a translation model and a reordering model.

[0068] like Figure 5 As shown, during inference, given a program x in the source language, the first step is to use a translation model to generate a set of candidate solutions using a bundle search method. Then, a reordering model is used to select the solution with the highest paraBLEU among the candidate solutions, which is taken as the final translation result of the program.

[0069] The following are system embodiments corresponding to the above method embodiments. This embodiment can be implemented in conjunction with the above embodiments. The relevant technical details mentioned in the above embodiments are still valid in this embodiment, and will not be repeated here to reduce repetition. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above embodiments.

[0070] This invention also proposes an intelligent programming language program translation system based on neural networks, which includes:

[0071] The initial module is used to construct a forward model that translates a source language program into a target language program, and a reverse model that translates a target language program into a source language program. Based on the source language program library and the target language program library, the forward model and the reverse model are trained by reverse translation to obtain the forward translation model and the reverse translation model.

[0072] The translation module is used to input the source language program to be translated into the forward translation model, input the multiple candidate results into the reordering model, obtain the score of each candidate result, and select the candidate result with the highest score as the translation result of the source language program.

[0073] The aforementioned neural network-based intelligent programming language program translation system, wherein the initial module is used for:

[0074] Select a program from the source language program library as the first training target and input it into the forward model to obtain the first target language. Input the intermediate target language into the reverse model to obtain the first source language. Construct the first loss based on the first source language and the first training target, and train the forward model and the reverse model.

[0075] Simultaneously, a program is selected from the target language program library as the second training target and input into the inverse model to obtain the second source language. The second source language is then input into the forward model to obtain the second target language. A second loss is constructed based on the second target language and the second training target, and the forward model and the inverse model are trained.

[0076] Until the first loss and the second loss converge, the current forward model and the backward model are saved as the forward translation model and the backward translation model, respectively.

[0077] The aforementioned neural network-based intelligent programming language program translation system, wherein the training of the reordering model includes:

[0078] Select program x from the target language program library and input it into the back translation model to generate an intermediate source program. Input the intermediate source program into the forward translation model to obtain multiple candidate results. Normalize the prediction score of the reordering model for each candidate result and obtain the output distribution.

[0079] Using the program x as a baseline, the correct score of each candidate result is obtained as the output distribution. The re-ranking model is trained by minimizing the divergence between the output distribution and the output distribution.

[0080] The aforementioned neural network-based intelligent programming language program translation system, wherein the source language is C, Python, or Java, and the target language is CUDA or BANG.

[0081] The present invention also proposes a storage medium for storing a program that executes any of the neural network-based intelligent programming language program translation methods.

[0082] The present invention also proposes a client for any of the aforementioned neural network-based intelligent programming language program translation systems.

Claims

1. A method for translating intelligent programming language programs based on neural networks, characterized in that, include: Step 1: Construct a forward model for translating a source language program into a target language program, and a reverse model for translating a target language program into a source language program; Based on the source language library and the target language library, the forward model and the reverse model are trained by reverse translation to obtain the forward translation model and the reverse translation model. Step 2: Input the source language program to be translated into the forward translation model, input the multiple candidate results into the reordering model, obtain the score of each candidate result, and select the candidate result with the highest score as the translation result of the source language program. Step 1 includes: Select a program from the source language program library as the first training target and input it into the forward model to obtain the first target language. Input the first target language into the reverse model to obtain the first source language. Construct a first loss based on the first source language and the first training target, and train the forward model and the reverse model. Simultaneously, a program is selected from the target language program library as the second training target and input into the inverse model to obtain the second source language. The second source language is then input into the forward model to obtain the second target language. A second loss is constructed based on the second target language and the second training target, and the forward model and the inverse model are trained. Until the first loss and the second loss converge, the current forward model and the backward model are saved as the forward translation model and the backward translation model, respectively. The training of this reordering model includes: Select program x from the target language program library and input it into the back translation model to generate an intermediate source program. Input the intermediate source program into the forward translation model to obtain multiple candidate results. Normalize the prediction score of the reordering model for each candidate result and obtain the output distribution. Using the program x as a baseline, the correct score of each candidate result is obtained as the output distribution. The re-ranking model is trained by minimizing the divergence between the output distribution and the output distribution.

2. The neural network-based intelligent programming language program translation method as described in claim 1, characterized in that, The source language is C, Python, or Java, and the target language is CUDA or BANG.

3. A neural network-based intelligent programming language program translation system, characterized in that, include: The initial module is used to build a forward model that translates a source language program into a target language program, and a reverse model that translates a target language program into a source language program. Based on the source language library and the target language library, the forward model and the reverse model are trained by reverse translation to obtain the forward translation model and the reverse translation model. The translation module is used to input the source language program to be translated into the forward translation model, input the multiple candidate results into the reordering model, obtain the score of each candidate result, and select the candidate result with the highest score as the translation result of the source language program. The initial module is used for: Select a program from the source language program library as the first training target and input it into the forward model to obtain the first target language. Input the first target language into the reverse model to obtain the first source language. Construct a first loss based on the first source language and the first training target, and train the forward model and the reverse model. Simultaneously, a program is selected from the target language program library as the second training target and input into the inverse model to obtain the second source language. The second source language is then input into the forward model to obtain the second target language. A second loss is constructed based on the second target language and the second training target, and the forward model and the inverse model are trained. Until the first loss and the second loss converge, the current forward model and backward model are saved as the forward translation model and the backward translation model, respectively. The training of this reordering model includes: Select program x from the target language program library and input it into the back translation model to generate an intermediate source program. Input the intermediate source program into the forward translation model to obtain multiple candidate results. Normalize the prediction score of the reordering model for each candidate result and obtain the output distribution. Using the program x as a baseline, the correct score of each candidate result is obtained as the output distribution. The re-ranking model is trained by minimizing the divergence between the output distribution and the output distribution.

4. The neural network-based intelligent programming language program translation system as described in claim 3, characterized in that, The source language is C, Python, or Java, and the target language is CUDA or BANG.

5. A storage medium for storing a program that executes the neural network-based intelligent programming language program translation method as described in claim 1 or 2.

6. A client that uses the neural network-based intelligent programming language program translation system as described in claim 3 or 4.

Images