AST processing method, neural network model training method, device and storage medium

Abstract

This invention discloses an AST processing method, a neural network model training method, an apparatus, and a storage medium. The AST processing method includes: obtaining an abstract syntax tree (AST) to be processed; inputting the AST into a trained neural network model, and embedding and encoding all nodes of the AST through a tree-based embedding layer in the neural network model to obtain an embedded and encoded AST; processing the embedded and encoded AST through an encoder in the neural network model to obtain an encoder-processed AST. The encoder is constructed based on a bidirectional neural network, which ensures that the paths of sibling nodes in the encoder-processed AST are different. The entire process can maintain the structured information of the original tree structure, thereby reducing the loss of original information; and processing the encoder-processed AST through a decoder in the neural network model to predict the target AST, thereby realizing an end-to-end conversion between the AST and the target AST.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to an AST processing method, a neural network model training method, an apparatus, and a storage medium. Background Technology

[0002] An Abstract Syntax Tree (AST) is an intermediate representation of a language that can be used for neural network analysis. An AST is a tree structure that reflects the types of nodes and the relationships between them.

[0003] In related technologies, when processing languages ​​such as programming languages ​​and natural languages, the AST of the original language is usually converted into a node sequence, and then the node sequence is input into a neural network model for processing.

[0004] However, since the node sequence input to the neural network model does not have the structured information of the original tree structure, the original information is greatly lost, resulting in lower accuracy of language processing. Summary of the Invention

[0005] In view of this, the main objective of the present invention is to provide an AST processing method, a neural network model training method, an apparatus, and a storage medium.

[0006] The technical solution of this invention is implemented as follows:

[0007] Firstly, an AST processing method is provided, the method comprising:

[0008] Obtain the Abstract Syntax Tree (AST) to be processed;

[0009] The AST is input into a trained neural network model, and all nodes of the AST are embedded and encoded through a tree-based embedding layer in the neural network model to obtain the embedded and encoded AST.

[0010] The AST after embedding is processed by the encoder in the neural network model to obtain the AST after encoder processing, wherein the encoder is constructed based on a bidirectional neural network;

[0011] The target AST is predicted by processing the AST processed by the encoder through the decoder in the neural network model.

[0012] Secondly, a method for training a neural network model is provided, the method comprising:

[0013] Obtain training samples, which include sample AST and sample target AST;

[0014] The training samples are input into a preset neural network model, and the tree-based embedding layer in the neural network model is used to embed and encode all nodes of the sample AST and all nodes of the sample target AST respectively.

[0015] The encoded sample AST is processed by the encoder in the neural network model to obtain the encoder-processed sample AST, wherein the encoder is constructed based on a bidirectional neural network;

[0016] The decoder in the neural network model processes the AST sample after the encoder and predicts the nodes of the target AST.

[0017] Based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST, the neural network model is trained using a forced-guided training mode.

[0018] Thirdly, an AST processing apparatus is provided, the apparatus comprising:

[0019] The acquisition module is used to acquire the Abstract Syntax Tree (AST) to be processed.

[0020] An embedding module is used to input the AST into a trained neural network model, and to embed and encode all nodes of the AST through a tree-based embedding layer in the neural network model to obtain the embedded and encoded AST.

[0021] An encoding module is used to process the embedded encoded AST through the encoder in the neural network model to obtain the encoder-processed AST, wherein the encoder is constructed based on a bidirectional neural network;

[0022] The decoding module is used to process the AST processed by the encoder through the decoder in the neural network model to predict the target AST.

[0023] Fourthly, a neural network model training device is provided, the device comprising:

[0024] The acquisition module is used to acquire training samples, which include the sample AST of the sample AST and the sample target AST.

[0025] An embedding module is used to input the training samples into a preset neural network model, and to embed and encode all nodes of the sample AST and all nodes of the sample target AST through a tree-based embedding layer in the neural network model.

[0026] An encoding module is used to process the embedded encoded sample AST through the encoder in the neural network model to obtain the encoder-processed sample AST, wherein the encoder is constructed based on a bidirectional neural network;

[0027] The decoding module is used to process the sample AST processed by the encoder through the decoder in the neural network model to predict the nodes of the target AST;

[0028] The training module is used to train the neural network model using a forced-guided training mode based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST.

[0029] Fifthly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of any of the AST processing methods described in the first aspect; or, the processor executes the program to implement the steps of any of the neural network model training methods described in the second aspect.

[0030] In a sixth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein when executed by a processor, the computer program implements the steps of any of the AST processing methods described in the first aspect; or, when executed by a processor, the computer program implements the steps of any of the neural network model training methods described in the second aspect.

[0031] The AST processing method, neural network model training method, apparatus, and storage medium provided in this invention embodiment are as follows: The AST processing method embeds and encodes all nodes of the abstract syntax tree (AST) to be processed through the tree-structured embedding layer in the neural network model to obtain the embedded and encoded AST; the encoder in the neural network model processes the embedded and encoded AST to obtain the encoder-processed AST. The encoder is constructed based on a bidirectional neural network, which makes the paths of sibling nodes in the encoder-processed AST different. The entire process can maintain the structured information of the original tree structure, thereby reducing the loss of original information; the decoder in the neural network model processes the encoder-processed AST to predict the target AST, realizing direct, end-to-end conversion between the AST and the target AST; and because the structured information of the AST is preserved during the analysis and processing of the AST by the neural network model, it is beneficial to obtain accurate prediction results and improve the accuracy of language processing. Attached Figure Description

[0032] Figure 1 This is a schematic flowchart of an AST processing method according to an embodiment of the present invention;

[0033] Figure 2 This is another flowchart illustrating the AST processing method according to an embodiment of the present invention;

[0034] Figure 3 This is another flowchart illustrating the AST processing method according to an embodiment of the present invention;

[0035] Figure 4 This is another flowchart illustrating the AST processing method according to an embodiment of the present invention.

[0036] Figure 5 This is a schematic diagram of the AST structure of the source code in an embodiment of the present invention;

[0037] Figure 6 This is a schematic diagram illustrating the process of embedding encoding using a tree-structured embedding layer according to an embodiment of the present invention;

[0038] Figure 7 This is a flowchart of the encoder performing a bottom-up traversal of the embedded encoded AST according to an embodiment of the present invention;

[0039] Figure 8 This is a flowchart of the encoder performing a top-down traversal of the AST after bottom-up traversal according to an embodiment of the present invention.

[0040] Figure 9 This is a schematic diagram of the encoder's bottom-up information transmission process according to an embodiment of the present invention;

[0041] Figure 10 This is a schematic diagram of the decoder processing flow according to an embodiment of the present invention;

[0042] Figure 11a-1 This is a partial structural diagram of the predicted target AST according to an embodiment of the present invention;

[0043] Figure 11a-2 This is a partial structural diagram of the predicted target AST according to an embodiment of the present invention;

[0044] Figure 11a-3 This is a partial structural diagram of the predicted target AST according to an embodiment of the present invention;

[0045] Figure 11b-1 This is a partial structural diagram of the actual target AST in an embodiment of the present invention;

[0046] Figure 11b-2 This is a partial structural diagram of the actual target AST in an embodiment of the present invention;

[0047] Figure 11b-3 This is a partial structural diagram of the actual target AST in an embodiment of the present invention;

[0048] Figure 12 This is a flowchart illustrating a neural network model training method according to an embodiment of the present invention;

[0049] Figure 13 This is a schematic diagram of the forced guided training process according to an embodiment of the present invention;

[0050] Figure 14 This is a schematic diagram of the AST processing device according to an embodiment of the present invention;

[0051] Figure 15 This is a schematic diagram of the structure of a neural network model training device according to an embodiment of the present invention;

[0052] Figure 16 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention. In the absence of conflict, the embodiments and features in the embodiments of this invention can be arbitrarily combined with each other. The steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowcharts, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0054] It is understood that the description of the various embodiments in this invention emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0055] In related technologies, when processing languages ​​such as programming languages ​​and natural languages, the original language's Abstract Syntax Tree (AST) is typically converted into a node sequence, which is then input into a neural network model for processing. However, in the process of converting the AST into a node sequence, existing search algorithms (such as depth-first search) cannot reflect the adjacency and sequential relationships between nodes at the same level. In other words, nodes at the same level may have multiple results after conversion into a sequence, meaning the converted sequence cannot retain the structural information of the original tree structure. This results in significant loss of original information, leading to lower accuracy in language processing.

[0056] In addition, among related technologies, decoders based on structured convolutional neural networks (CNNs) implement the use of CNNs for code generation, and tree-based CNNs are given. However, the encoder part is not provided, so it cannot complete the end-to-end task. Furthermore, such decoders require some additional information, such as program description information and function or method scope, which require additional processing and cannot directly and efficiently generate code.

[0057] Furthermore, due to the existence of Out Of Vocabulary (OOV) words in languages—that is, a large number of user-defined words, such as various variable names and function names defined in program code—it is impossible to count them all during the network training phase. Therefore, these words are usually uniformly encoded as Unknown Tokens (UNKs). If different OOVs are uniformly encoded as Unknown Tokens, it will lead to the inability to distinguish between different OOVs, affecting the accuracy of the prediction results. Moreover, in related technologies, after the neural network model analyzes and processes the AST, the output result is usually a classification representation, but the network that can generate the AST suffers from problems such as ignoring the processing of OOVs and unstable model convergence.

[0058] To address one or more technical problems existing in the aforementioned related technologies, a technical solution according to embodiments of the present invention is proposed. The technical solution of the embodiments of the present invention can be applied to language processing scenarios such as programming languages ​​and natural languages, for example, natural language understanding, natural language generation, code conversion and generation, such as the mutual conversion between source code and target code, the mutual conversion between natural language and programming languages, etc. The embodiments of the present invention do not specifically limit the specific language processing scenarios.

[0059] The following description uses a programming language processing scenario applied to code conversion as an example to illustrate the technical solution provided by the embodiments of the present invention.

[0060] Figure 1 A flowchart illustrating the AST processing method provided in this embodiment of the invention is shown below. Figure 1 The AST processing method of this invention includes the following steps:

[0061] 101. Obtain the Abstract Syntax Tree (AST) to be processed.

[0062] The abstract syntax tree (AST) to be processed can be the AST of the source code. It is understandable that the source of the AST to be processed differs in different language processing scenarios; that is, the AST to be processed is related to the specific language processing task.

[0063] Source code can be program code written in any type of language rules, and the language of the source code includes, but is not limited to, programming languages ​​and scripting languages. The Abstract Syntax Tree (AST) of source code refers to the tree structure corresponding to the syntax of the source code. For source code in a specific programming language, statements in the source code are mapped to each node in the tree by constructing a syntax tree. In other words, the AST of source code is a tree-like representation of the abstract syntactic structure of the source code, and each node in the tree represents a structure in the source code.

[0064] Here, the source code contains code for one or more functions, and the abstract syntax tree (AST) of the source code includes the AST corresponding to each function in the source code. Here, a function is a piece of program or code that can be directly referenced by another program or code. A function is used to implement a specific function.

[0065] Specifically, obtaining the Abstract Syntax Tree (AST) of the source code can include:

[0066] The source code is extracted using function declarations as the basic unit, resulting in each function in the source code. The source code is then parsed to obtain the AST corresponding to each function. A preorder traversal is performed on the AST corresponding to each function, and an ID value is assigned to each node in the AST. Generally, natural number encoding can be used to set the ID value, for example, 1 to M, where M represents the total number of different node names in the entire dataset. Each node obtains a different number as its encoding value.

[0067] This involves parsing the source code to obtain the Abstract Syntax Tree (AST) for each function, including:

[0068] If the source code being parsed is a programming language, the entire AST of the source code is obtained first, and then the subtrees defining function declarations are extracted as basic data units; if the source code being parsed is a scripting language, the functions contained in the source code are directly converted into the corresponding AST.

[0069] In some embodiments, the Abstract Syntax Tree (AST) is in binary tree form. Generally, AST refers to its general form, but in neural networks, it is converted into a binary tree structure for subsequent computation. In practice, the source code can be directly converted into a binary tree AST, or it can be first converted into a general AST and then converted back into a binary tree AST.

[0070] Understandably, when the source code is in C / C++, Java, or PHP, tools including Clang Lib can be used to perform AST transformation.

[0071] 102. Input the AST into the trained neural network model, and use the tree-based embedding layer in the neural network model to embed and encode all nodes of the AST to obtain the embedded and encoded AST.

[0072] The neural network model includes a tree-based embedding layer, an encoder, and a decoder. The encoder can be constructed from a bidirectional tree-based neural network, and the decoder can also be constructed from a tree-based neural network. These neural networks include, but are not limited to, those constructed from one or more of the following: Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) networks (an improved version of RNN), Transformer, and Gated Recurrent Unit (GRU). The training method for this neural network model will be described in detail below through another methodological embodiment.

[0073] Specifically, the AST is input into the trained neural network model, and all nodes of the AST are embedded and encoded by a tree-based embedding layer in a traversal manner to obtain the embedded and encoded AST. This traversal method can be a preorder traversal, that is, following a certain path and passing through all nodes in the order of root to left and right.

[0074] An Abstract Syntax Tree (AST) needs to be converted into quantified vectors for analysis and processing in neural network models, typically achieved through spatial embedding. In this embodiment, during the embedding encoding process of the AST input into a tree-based embedding layer, a traversal algorithm is used to control the order of node embedding encoding, ensuring that all nodes of the AST are embedded and encoded according to a tree structure. This allows the embedded AST to distinguish the type of each node and the relationships between nodes. Furthermore, by embedding and encoding all nodes of the AST through a tree-based embedding layer, the spatial structure and syntactic information of the AST can be converted into high-dimensional feature vectors for subsequent calculations.

[0075] 103. The AST after embedding is processed by the encoder in the neural network model to obtain the AST after encoder processing. The encoder is constructed based on a bidirectional neural network.

[0076] The encoder can use a deep bidirectional tree-based neural network to traverse and update the embedded encoded AST.

[0077] Specifically, the encoder can first pass information from leaf nodes to root nodes from bottom to top after embedding the encoded AST, and finally gather it to the root node of the whole tree. Then, the feature information of the AST is passed from root node to leaf nodes from top to bottom.

[0078] 104. The target AST is predicted by processing the encoder-processed AST through the decoder in the neural network model.

[0079] The decoder can use a tree-based neural network to process the AST processed by the encoder and predict the target AST.

[0080] Specifically, the decoder uses an autoregressive decoding method to predict and expand the root nodes of the AST processed by the encoder, passing information from the root nodes to the leaf nodes from top to bottom. In practice, the decoder can recursively predict whether the current node has child nodes using basic computational units. If it does, it predicts which nodes have the highest probability of being the left and right child nodes. Once a node is predicted, it continues to predict and expand its left and right child nodes until all nodes no longer have child nodes. The target AST is obtained based on the hierarchical relationship between the traversed nodes.

[0081] In some embodiments, when the AST is the AST of the source code, the target AST is used to transform and generate target code. The target code can then be used for driver code generation.

[0082] Specifically, the target AST is directly converted into target code, without needing to merge the obtained target AST into a whole tree as a subtree before converting it into target code.

[0083] In the aforementioned AST processing method, all nodes of the Abstract Syntax Tree (AST) of the source code are embedded and encoded through a tree-structured embedding layer in the neural network model to obtain the embedded and encoded AST. The encoder in the neural network model then processes the embedded and encoded AST to obtain the encoder-processed AST. The encoder is constructed based on a bidirectional neural network, ensuring that the paths of sibling nodes in the encoder-processed AST are different. The entire process maintains the structured information of the original tree structure, thereby reducing the loss of original information. The decoder in the neural network model then processes the encoder-processed AST to predict the target AST, thus achieving end-to-end conversion between the AST and the target AST. Furthermore, since the structured information of the AST is preserved during the analysis and processing of the AST by the neural network model, it is beneficial to obtain accurate prediction results. This method can be applied to various language processing scenarios, thereby improving the accuracy of language processing.

[0084] In some embodiments, step 102 above, which involves embedding and encoding all nodes of the AST through a tree-based embedding layer in a neural network model to obtain the embedded and encoded AST, may include:

[0085] When the node name of the AST node is not the dictionary overflow word OOV, obtain the embedding code of the AST node based on the ID value of the AST node.

[0086] When the node name of an AST node is a dictionary overflow word OOV, the embedding encoding of the AST node is obtained from the natural distribution of the weight matrix of the embedding layer.

[0087] Specifically, the node name of the AST node is determined based on the preset vocabulary (i.e., the dictionary) to determine whether it is a dictionary overflow word OOV.

[0088] The process of obtaining the embedding encoding of nodes in the AST based on their ID values ​​includes:

[0089] Based on the ID value of the node in the AST, the node in the AST is one-hot encoded, and the one-hot encoding is multiplied by the weight matrix of the embedding layer. The resulting vector is used as the embedding encoding of the node in the AST.

[0090] The embedding encoding of nodes in the AST is obtained from the natural distribution of the weight matrix of the embedding layer, including:

[0091] Based on the mean and variance of the weight matrix of the embedding layer, a natural distribution is generated, and the embedding encoding of the nodes of the AST is determined according to the vector generated from the natural distribution. Specifically, a vector can be randomly generated from the natural distribution as the embedding encoding of the nodes of the AST.

[0092] One-hot encoding, also known as single-bit encoding, uses an N-bit state register to encode N states. Each state has its own independent register bit, and at any given time, only one bit is active. For example, to encode six states, the natural sequence code is 0, 1, 2, 3, 4, 5, and the one-hot code is 000001, 000010, 000100, 001000, 010000, 100000.

[0093] In this embodiment, the embedding encoding process for nodes with the name OOV is as follows: a natural distribution with the same mean and variance as the weight matrix is ​​constructed, and a vector with the same number of columns as the weight matrix is ​​randomly generated as its embedding encoding. The embedding encoding process for nodes with names other than OOV (i.e., not overflow words) is as follows: the node ID value (a single number) is converted to a one-hot encoding, then the one-hot encoding is multiplied by the weight matrix, and the resulting vector is used as the embedding encoding. Since the one-hot encoding has a value of 1 in only one bit and 0 in other bits, this means that the embedding encoding obtained after multiplying the one-hot encoding by the weight matrix is ​​a row of the weight matrix.

[0094] As can be seen, tree-based embedding layers in neural network models allow the embedding encoding mapping process to occur throughout the entire domain, while also differentiating OOVs in their embedding encoding: General embedding encoding transforms the input from a single numerical value into a one-hot encoding, which is then multiplied by a weight matrix. This means that the total number of rows or columns in the weight matrix determines all possible outputs of the embedding layer. However, the sparsity of one-hot encoding means the weight matrix is ​​not fully utilized. By utilizing the mean and variance of the weight matrix to generate a natural distribution, and then generating a vector from this natural distribution as the embedding encoding for the OOV, the embedding encoding of all nodes in the tree structure follows the same distribution, and OOVs with the same name on the same tree also have the same output.

[0095] In some embodiments, see Figure 2 In step 103 above, the encoder in the neural network model processes the embedded AST to obtain the encoder-processed AST, which may include:

[0096] 1031, The encoder transmits information from leaf nodes to root nodes from bottom to top through the embedded AST;

[0097] 1032. The encoder performs information transfer from the root node to the leaf node from top to bottom on the AST after the bottom-up information transfer is completed, and obtains the encoder-processed AST.

[0098] The encoder can be constructed from a deep bidirectional tree-based neural network, including but not limited to bidirectional RNN, bidirectional LSTM, and bidirectional GRU.

[0099] Specifically, the encoder performs information transfer in two directions on the embedded encoded AST based on a deep bidirectional tree-structured neural network. One direction of information transfer is from the bottom up from the leaf nodes to the root node of the embedded encoded AST. The other direction of information transfer is from the root node to the leaf nodes of the AST after the bottom-up information transfer is completed. After the bottom-up information transfer is completed, the AST processed by the encoder is obtained.

[0100] In this embodiment, in addition to the bottom-up information transmission process that gathers all the information of the entire AST tree at the root node, a top-down information transmission process is also used to update all nodes of the AST after the bottom-up information transmission is completed. This not only preserves the structured information of the original tree structure, but also, compared to the unidirectional information transmission method, enables the identification of nodes and features under the same path. This solves the problem that nodes at the same level may have the same path, and suppresses the situation where the target AST result is not unique due to ambiguity when the subsequent decoder traverses the nodes, thereby improving the accuracy of the prediction results.

[0101] In some embodiments, step 1031 above, which involves transmitting information from leaf nodes to root nodes in the embedded AST from bottom to top using an encoder, may include:

[0102] 1031a, the encoder traverses each node of the embedded encoded AST and takes the currently traversed node as the current root node;

[0103] 1031b: Based on the hidden states of the left and right child nodes of the current root node and the embedding encoding of the current root node, update the hidden state of the current root node until the feature information of the embedded AST is gathered into the root node of the whole tree, and then end the traversal.

[0104] Specifically, the encoder can use a preorder traversal. Each time a node is entered, the left and right child nodes of the current node are traversed first, and then the hidden state of the current node is calculated. Since the child nodes of the current node are traversed first, the hidden states of the child nodes of the current node are updated first, and then the hidden state of the current node is updated, until the root node containing the feature information of the embedded encoded AST is obtained. The traversal ends, thus completing the bottom-up information transfer.

[0105] It is understandable that, in addition to preorder traversal, other traversal methods can be used, as long as the bottom-up node update rule is satisfied. This embodiment does not limit this.

[0106] In some embodiments, step 1032, where the encoder performs a top-down information transfer from the root node to the leaf node on the AST after the bottom-up information transfer is completed, to obtain the encoder-processed AST, may include:

[0107] 1032a, by traversing each node of the AST after the bottom-up information transmission is completed by the encoder, the currently traversed node is taken as the current root node;

[0108] 1032b updates the hidden state of the current root node based on the hidden state of the parent node of the current root node, the hidden state of the sibling node of the current root node, and the embedding encoding of the current root node until the traversal ends, and obtains the AST processed by the encoder. The sibling node of the current root node has the same parent node as the current root node.

[0109] In this embodiment, when the encoder processes the hidden state of a node after the bottom-up information transmission is completed, the left child node will be updated before the sibling node (i.e., the right child node of the parent node), and obtain the updated parent node information and the unupdated sibling node information, while the right child node obtains the updated parent node information and the updated sibling node information.

[0110] In some embodiments, the encoder is constructed based on a deep bidirectional tree-based long short-term memory neural network (Deep Bidirectional Tree-LSTMs).

[0111] Here, similar to the standard LSTM structure, each cell in Tree-LSTM includes a similar input gate i_t, output gate o_t, cell state c_t, and hidden layer output h_t. The number of gates in Standard LSTM and Tree-LSTM is exactly the same; the difference lies in their computation methods. Standard LSTM computes information sequentially, such as passing information from the previous token to the next token during forward traversal, while Tree-LSTM computes information during tree traversal, such as passing information from child nodes to parent nodes during bottom-up traversal. In this embodiment, the encoder uses a deep bidirectional tree-based Long Short-Term Memory (LSTM) neural network. Compared to other neural network structures, using a deep bidirectional tree-based LSTM network in the encoder part provides better feature capture capabilities.

[0112] In some embodiments, step 104, processing the encoder-processed AST using the decoder in the neural network model to predict the target AST, may include:

[0113] Based on the AST processed by the encoder, the left and right child nodes of each node in the target AST are predicted by the decoder using an autoregressive decoding method, thus obtaining the target AST.

[0114] Specifically, for the AST processed by the encoder, the decoder can calculate the hidden state of the nodes using a tree-based long short-term memory network, and predict the left and right child nodes of the root node of the AST corresponding to the copied OOV using a pointer network or a copy network. For the AST corresponding to codes other than OOV, the left and right child nodes of the root node of the AST can be predicted using an MLP (Multi-Layer Perceptron).

[0115] Pointer networks can be used to directly copy certain elements from the input sequence to the output sequence. MLP is a common ANN (Artificial Neuro Network) algorithm, consisting of an input layer, an output layer, and one or more hidden layers, outputting a tensor representing the generation probabilities of all words in the vocabulary. Here, a tensor is a data container, typically containing numerical data.

[0116] Generally, Out-of-Void (OOV) errors often occur in leaf nodes. After the encoder passes information from leaf nodes to the root node in the AST from bottom to top, most OOV encoding results are the same. This means that when the decoder calculates the attention vector, they will all have the same feature values, resulting in the same copy probability. However, in this embodiment, the encoder performs bidirectional information transmission on the embedded AST, which can distinguish the features of all nodes, including OOV errors. Thus, by predicting and copying the left and right child nodes of the AST corresponding to the OOV during the decoding process, the decoder can suppress the data loss that may occur during code generation due to the same copy probability of OOV errors. This solves the problem of increased convergence difficulty caused by information loss due to the presence of OOV errors and is conducive to obtaining more accurate prediction results.

[0117] In some embodiments, the decoder is constructed based on a tree-based decoding neural network, a pointer network, and a multilayer perceptron. The neural network can be an RNN, LSTM, or GRU, etc.

[0118] See Figure 3 The AST processed by the encoder is then used by the decoder to predict the left and right child nodes of each node in the target AST using an autoregressive decoding method, thus obtaining the target AST. This process may include the following steps:

[0119] 1041. The root node of the AST processed by the encoder is used as the initial node of the target AST and input into the tree-based neural network.

[0120] 1042. For the node input into the tree-based neural network at the current time, calculate the hidden states of the left and right child nodes of the node using the tree-based neural network.

[0121] 1043. Based on the AST processed by the encoder and the hidden states of the left and right child nodes of the node, the left and right child nodes of the replica node are predicted by the pointer network to obtain the replication probability and the first prediction result.

[0122] 1044. Based on the hidden states of the left and right child nodes of a node, the left and right child nodes of the generated node are predicted using a multilayer perceptron to obtain the second prediction result.

[0123] 1045. Based on the replication probability, the first prediction result and the second prediction result are weighted and merged to generate the left child node and the right child node of the node. The generated left child node and the right child node are used as the nodes of the tree-based neural network in the next time step.

[0124] 1046. Repeat steps 1042 to 1045 until all leaf nodes of the target AST are generated, and then stop to obtain the target AST.

[0125] The first prediction result includes a tensor of the probability that all nodes of the AST processed by the encoder may be copied; the second prediction result includes a tensor of the generation probability of all words in the preset vocabulary.

[0126] In this case, the sum of the probabilities in a tensor included in the first prediction result and the second prediction result is 1. That is, in the tensor of the probability of all nodes in the encoder-processed AST included in the first prediction result, the sum of the probabilities of all nodes being copied is 1. In the tensor of the generation probability of all words in the preset vocabulary included in the second prediction result, the sum of the probabilities of all words being generated is also 1.

[0127] Since the predicted child node name has only two possibilities (OOV and non-OOV), the sum of the probability of the pointer network predicting the replication of a child node and the probability of the multilayer perceptron predicting the generation of that child node is 1. In other words, after obtaining the replication probability P output by the pointer network for predicting the replication of a child node, we can obtain the generation probability (1-P) of the multilayer perceptron for predicting the generation of that child node. Here, the replication probability represents the probability of the child node being predicted to be replicated from the AST processed by the encoder, and the generation probability represents the probability of the child node being predicted to be generated from the predefined vocabulary (i.e., the dictionary).

[0128] Specifically, based on the replication probability of a child node of a replicated node and the generation probability determined according to the replication probability, a tensor for predicting the replication of the child node included in the first prediction result and a tensor for predicting the generation of the child node included in the second prediction result are weighted and merged, and the child node of the node is generated according to the weighted merging result.

[0129] More specifically, when predicting the left child of a node, the sum of the probability of the pointer network predicting replication of the left child and the probability of the multilayer perceptron predicting generation of the left child is 1. After obtaining the replication probability of the pointer network indicating the left child, the generation probability of the multilayer perceptron indicating the left child can be determined based on this replication probability. Then, based on this replication probability and the determined generation probability, the tensors indicating the left child in the first prediction result output by the pointer network and the tensors indicating the left child in the second prediction result output by the multilayer perceptron are weighted and merged. Based on the weighted merging result, the node with the highest prediction probability is determined, and this left child is the node with the highest prediction probability.

[0130] Similarly, when predicting the right child node of a certain node, the sum of the probability of the pointer network predicting the replication of the right child node and the probability of the multilayer perceptron predicting the generation of the right child node is also 1. Therefore, after obtaining the replication probability of the pointer network indicating the right child node, the generation probability of the multilayer perceptron predicting the generation of the right child node can be determined based on the replication probability. Then, based on the replication probability and the determined generation probability, the tensor indicating the right child node in the first prediction result output by the pointer network and the tensor indicating the right child node in the second prediction result output by the multilayer perceptron are weighted and merged. Based on the weighted merging result, the node with the highest prediction probability is determined, and the right child node is the node with the highest prediction probability.

[0131] For ease of description, the tensor in the first result that contains the probability of all nodes in the encoder-processed AST being copied is denoted as the first tensor, and the tensor in the second prediction result that contains the probability of all words in the preset vocabulary being generated is denoted as the second tensor.

[0132] The aforementioned weighted merging process includes:

[0133] The replication probability and the generation probability determined based on the replication probability are used as the weights of the first tensor and the second tensor, respectively. Based on whether the node names are the same, the probabilities in the first tensor and the probabilities in the second tensor are weighted and summed according to the same node names, and a tensor is generated based on the weighted summation result.

[0134] For example, assume that all nodes in the AST processed by the encoder are I(a, e, f); and all words in the dictionary are V(a, b, c, d, e). The left child of a node n is predicted and copied using a pointer network, resulting in the copy probability p indicating the left child of node n, and a first prediction result. This first prediction result includes a tensor Y representing the probability of all nodes in the encoder-processed AST being copied. I The left child node of node n is predicted using a multilayer perceptron, resulting in a second prediction that indicates the left child node of node n. This second prediction includes a tensor Y representing the generation probability of all words in a predefined vocabulary. V Wherein: the replication probability p is 0.9, and the tensor Y... I = (0.8, 0.1, 0.1). Tensor Y V (0.9, 0.025, 0.025, 0.025, 0.025). Using the replication probability p, the generation probability (1-p) of the multilayer perceptron is calculated to be 0.1. Based on the replication probability p of the pointer network and the generation probability (1-p) of the multilayer perceptron, the tensor Y is... I and tensor Y V We perform a weighted merging and obtain the predicted probabilities of a to f based on the weighted merging results. The predicted probability of a is 0.9*0.8+0.1*0.9=0.81, which is the highest. Therefore, we determine a as the left child node of node n.

[0135] For the child nodes of the node predicted by the decoder, regardless of whether the child node is a left child node or a right child node, there are three possible cases:

[0136] a. This child node is a node in the AST after the encoder has processed it, and the node name of this child node is also included in the dictionary;

[0137] b. The node name of the child node is included in the dictionary, but the child node is not a node in the AST processed by the encoder.

[0138] c. This child node is a node in the AST after the encoder has processed it, but the node name of this child node is not included in the dictionary.

[0139] The decoder, based on the replication probability, weights and merges the first and second prediction results to generate the child nodes of a node. It can determine whether a node's child node is generated from the vocabulary or copied from the input AST. The vocabulary, a pre-defined lexicon, is used to predict and generate non-OOV (Out-of-Vocabulary) nodes. The dictionary uses a key-value pair structure to record the node ID corresponding to the non-OOV node and the node information corresponding to that ID, including but not limited to function names, function declarations, and variable names.

[0140] In this embodiment, since user-defined words in the language, such as variable names and function names in program code, are inevitably unpredictable, it is impossible to generate them probabilistically. However, probabilistic copying can perfectly solve this problem. By introducing a pointer network into the decoder, the copying probability can be calculated more accurately for all OOVs in the input, solving the problem of information loss caused by the presence of OOVs, which increases the difficulty of convergence and thus helps to obtain more accurate prediction results.

[0141] In some embodiments, the tree-based neural network described above is a tree-based Long Short-Term Memory (LSTM) neural network. This LSTM neural network includes: a left LSTM network for computing the hidden states of the left child nodes of a node, and a right LSTM network for computing the hidden states of the right child nodes of a node. The left and right LSTM networks each contain a set of basic computational units (LSTM Cells). Correspondingly, the decoder includes: a left pointer network and a left multilayer perceptron connected to the left LSTM network, and a right pointer network and a right multilayer perceptron connected to the right LSTM network.

[0142] The following is combined with Figures 4 to 1 1b. Continuing with the example of a programming language processing scenario applied to code conversion, the AST processing method of this embodiment of the invention will be further explained.

[0143] Figure 4 This is a flowchart illustrating the AST processing method according to an embodiment of the present invention. Figure 4In this diagram, lines of varying lengths represent code, binary trees represent the generated AST (Abstract Syntax Tree), circles represent nodes in the hidden state, and cylinders represent nodes in the updated state. Circles and cylinders filled with a grid pattern represent the root node of the entire AST tree, circles and cylinders filled with a diagonal pattern represent leaf nodes, and circles and cylinders without line filling represent general nodes. Black rounded rectangles represent the layer structure of the neural network. "Tree-Lstm+number" indicates the layer of the neural network corresponding to the encoder. "Lstm-cell+number" indicates the layer of the neural network corresponding to the decoder. Numbers range from 1 to N, indicated by ellipses, representing N layers, where N is a variable value.

[0144] like Figure 4 As shown, the AST processing method may include: extracting the AST from the source code to obtain the corresponding AST; inputting the obtained AST into a tree-based embedding layer for traversal and outputting the embedded and encoded AST; inputting the embedded and encoded AST into an encoder, performing feature learning using deep bidirectional tree-based LSTMs, and outputting the encoder-processed AST; the decoder using deep LSTMs to predict the input encoder-processed AST and outputting the target AST; and finally, converting the target AST into the driving code corresponding to the source code.

[0145] Assuming the driver generation rule is: initialize each parameter of the input source code (function) using the corresponding random function, and then call it as a function parameter, the AST processing method of this embodiment of the invention will be illustrated using Python source code as an example. For example, assume the Python source code is as follows:

[0146] def eoqcu(iimztd):

[0147] …

[0148] Perform the following steps (1) to (5) on the above Python source code to generate the driver code.

[0149] (1) Extracting the AST from the source code

[0150] Specifically, the source code can be directly converted into a binary tree AST, or the source code can be first converted into a general AST, and then the general AST can be converted into a binary tree AST.

[0151] Extracting the AST from the Python source code of the above example yields... Figure 5 The binary tree form of the AST. For example... Figure 5As shown, circles and ellipses represent nodes, and the text labeled within them is the node name. The number in parentheses below the node name indicates the traversal sequence, that is, which node was visited during the entire traversal.

[0152] (2) Tree-Embedding Layer is used to embed and encode the AST.

[0153] The obtained AST is input into a tree-based embedding layer for traversal, and the embedded AST is output. In practice, before entering the tree-embedding layer, the AST is first traversed in preorder, and an ID value is set for each node.

[0154] Figure 6 This is a schematic diagram illustrating the embedding encoding process using a tree-structured embedding layer, as described in an embodiment of the present invention. Figure 6 As shown, the Tree-Embedding Layer performs embedding encoding on the input AST using a recursive traversal approach as follows:

[0155] 1. Start by entering the nodes of the AST.

[0156] 2. Set the entry node as the root node.

[0157] 3. Determine if the root node's name is OOV: If it is, obtain the root node's embedding code from the natural distribution of the weight matrix; if not, obtain the root node's embedding code based on the node ID value.

[0158] 4. Check if the root node with embedded encoding has a left child node.

[0159] 5. If a left child node exists, proceed to the left child node and check if the root node with embedded encoding has a right child node.

[0160] 6. If there is no left child node, check if there is a right child node in the root node that has been embedded and encoded.

[0161] 7. If a right child node exists, proceed to the right child node.

[0162] 8. After entering the left child node and the right child node, execute steps 2 to 7 respectively.

[0163] 9. If there is no right child node, the embedding encoding ends.

[0164] The embedding encoding process for a node named OOV involves constructing a natural distribution with the same mean and variance as the weight matrix, and randomly generating a vector with the same number of columns as the weight matrix as its embedding encoding.

[0165] The embedding encoding process for nodes whose names are not OOV involves converting the node's ID value (a single number) into a one-hot code, multiplying it by the weight matrix, and using the resulting vector as the embedding code. Since the one-hot code has a value of 1 in only one bit and 0 in the others, the embedding code obtained after matrix multiplication is a row of the weight matrix.

[0166] (3) The encoder processes the embedded encoded AST.

[0167] The embedded encoded AST is input into the encoder, and deep bidirectional tree-based LSTMs are used for feature learning to output the encoder-processed AST. The specific implementation process includes: the encoder traversing the embedded encoded AST from bottom to top, and the encoder traversing the AST from top to bottom after the bottom-up traversal.

[0168] Figure 7 This is a flowchart illustrating the bottom-up traversal of the embedded AST by the encoder in an embodiment of the present invention. Figure 7 As shown, the encoder's bottom-up traversal process for the AST after bottom-up traversal is as follows:

[0169] 1. Begin.

[0170] 2. Initialize queue 1 and queue 2, and add the root node to queue 1.

[0171] 3. Take a node from queue 1, use it as the current root node, and add it to queue 2.

[0172] 4. Check if a left child node exists. If it does, add the left child node to queue 1 and continue to check if a right child node exists.

[0173] 5. If there is no left child node, then check if there is a right child node.

[0174] 6. If a right child node exists, add the right child node to queue 1, and continue to check whether queue 1 is empty.

[0175] 7. If there is no right child node, check if queue 1 is empty.

[0176] 8. If queue 1 is not empty, repeat steps 3 to 7.

[0177] 9. If queue 1 is empty, then take a node from queue 2 as the current root node.

[0178] 10. Determine if a left child node exists. If no left child node exists, initialize the hidden state of the left child node to a tensor of all zeros, and then determine if a right child node exists.

[0179] 11. If a left child node exists, then continue to check if a right child node exists.

[0180] 12. If there is no right child node, initialize the hidden state of the right child node to a tensor of all zeros, and calculate and update the hidden state of the root node based on the hidden states of the left and right child nodes.

[0181] 13. If a right child node exists, calculate and update the hidden state of the root node based on the hidden states of the left and right child nodes.

[0182] 14. Determine if queue 2 is empty.

[0183] 15. If queue 2 is empty, then end the traversal.

[0184] 16. If queue 2 is not empty, repeat steps 9 to 13.

[0185] The encoder can use a preorder traversal. Upon entering a node, it first traverses the node's left and right children, then calculates the hidden state of the current node. Because the child nodes are traversed first, their hidden states are updated first, thus completing bottom-up information transfer. Compared to recursive traversal, the encoder uses a queue traversal, which avoids overflow and is more efficient. Of course, other traversal methods besides preorder traversal can be used, as long as they satisfy the bottom-up node update rule; no specific restrictions are imposed here.

[0186] Figure 8 This is a flowchart illustrating the encoder's top-down traversal of the AST after bottom-up traversal, according to an embodiment of the present invention. Figure 8 As shown, the encoder's top-down traversal process for the AST after bottom-up traversal is as follows:

[0187] 1. Begin.

[0188] 2. Initialize the queue and add the root node to the queue.

[0189] 3. Take a node from the queue and use it as the current root node.

[0190] 4. Determine if a parent node exists.

[0191] 5. If there is no parent node, initialize the hidden state of the parent node and sibling nodes to all-zero tensors, and calculate and update the hidden state of the root node based on the hidden states of the parent node and sibling nodes.

[0192] 6. If a parent node exists, retrieve the parent node of the current root node.

[0193] 7. Determine if the root node is the left child of its parent node.

[0194] 8. If the root node is not the left child of the parent node, then obtain the right child of the parent node as the sibling node of the root node, and calculate and update the hidden state of the root node based on the hidden states of the parent node and the sibling node.

[0195] 9. If the root node is the left child of the parent node, then obtain the right child of the parent node as the sibling node of the root node, and calculate and update the hidden state of the root node based on the hidden states of the parent node and the sibling node.

[0196] 10. Determine if the current root node has a left child node. If it does, add the left child node to the queue. Continue to determine if the current root node has a right child node. If it does, add the right child node to the queue.

[0197] 11. Check if the queue is empty.

[0198] 12. If the queue is not empty, proceed to steps 3 through 11.

[0199] 13. If empty, end the traversal.

[0200] During the encoder's processing of a node's hidden state, the left child node updates before its sibling node (i.e., the right child node of the parent node), obtaining the updated parent node information and the unupdated sibling node information, while the right child node obtains the updated parent node information and the updated sibling node information.

[0201] Figure 9 This is a schematic diagram of the bottom-up information transmission process of the encoder according to an embodiment of the present invention. The embedded AST is input into the encoder and calculated according to the timing sequence. Figure 9 As shown in the diagram, the markings on the cylinders in the AST schematic represent the temporal sequence of the traversal process. Taking time t8 as an example, assuming the root node x has 300 features, the left and right child nodes both have a feature value of child_h of 150, and the memory feature value child_c is 150, these feature values ​​can be understood as the dimension of the feature vector. The rectangle in the right figure represents one layer of the Tree-LSTM neural network corresponding to the encoder (i.e.,...). Figure 4 The diagram shown illustrates how the encoder (corresponding to one of the N layers in a neural network) calculates the feature value x_h and memorizes the feature value x_c of the root node x. N represents the number of child nodes of a node; for a binary tree, N is 2.

[0202] In this context, characters starting with "w_" represent weight matrices. `w_fh` represents the weight matrix mapping to the child node eigenvalue `child_h`, and `w_fx` represents the weight matrix mapping to the root node `x`. `fh` represents the result of matrix transformation on the child node eigenvalue `child_h`, and `fx` represents the result of matrix transformation on the root node `x`. `f` represents the sum of the matrix transformations on the root node `x` and the child node eigenvalue `child_h`.

[0203] fc represents the matrix multiplication result of f with the memory eigenvalue child_c. fc_sum represents the summation of fc along the 0th dimension. i, o, u represent the three gate structures in the basic unit of LSTM. w_x represents the weight matrix for calculating the IoU result with respect to the root node x, iou_x represents the output of the three gate structures when the root node x is the input, w_h represents the weight matrix for calculating the IoU result with respect to the child node eigenvalue child_h, iou_h represents the output of the three gate structures when the child node is the input, x_c represents the currently updated memory eigenvalue of the root node x, and x_h represents the currently updated eigenvalue of the root node x.

[0204] The sigmoid function, also known as the logistic function, is used for the output of hidden layer neurons. Its value ranges from (0, 1), mapping a real number to the interval (0, 1), and is used for binary classification. It performs well when the features differ significantly or not by a large margin. The tanh function returns the hyperbolic tangent of any real number.

[0205] (4) The decoder processes the AST after it has been processed by the encoder.

[0206] The decoder uses tree-based deep LSTMs to predict the input AST processed by the encoder and outputs the target AST.

[0207] Figure 10 This is a schematic diagram of the decoder processing flow according to an embodiment of the present invention. Figure 10As shown, the AST output by the encoder is input to the decoder. The decoder uses an autoregressive decoding method to predict the left and right child nodes of the generated or copied root node, outputting the target AST. The AST is then converted into code to obtain the driving code. The neural networks used to calculate the predicted left and right child nodes are represented by a set of Left Lstm Cells (basic computational units of the left long short-term memory network) and Right Lstm Cells (basic computational units of the right long short-term memory network), respectively. Each set consists of N cells concatenated and numbered with underscores. They are used to calculate the hidden states of nodes. The Pointer Network generates a copy probability P and a tensor representing the probability of all input nodes being copied. The MLP (Multi-Layer Perceptron) outputs a tensor representing the generation probability of all words in the vocabulary.

[0208] The process by which the decoder predicts or generates the left and right child nodes of the root node is as follows:

[0209] 1. Initially, the root node of the input AST is used as the initial node of the decoder.

[0210] 2. Calculate the hidden state of the left child node using Left Lstm Cells.

[0211] 3. Based on the hidden state of the left child node, the pointer network generates a copy probability P and a tensor of the possible copy probabilities of all input nodes.

[0212] 4. Based on the replication probability P and its tensor generated by the pointer network, combined with the tensor of the generation probability of all words in the vocabulary output by the multilayer perceptron, the left child node is predicted, and the hidden state of the left child node is used as the input of the next time step of Left LstmCells.

[0213] 5. Calculate the hidden state of the right child node using Right Lstm Cells.

[0214] 6. Based on the hidden state of the right child node, the pointer network generates a copy probability P and a tensor of the possible copy probabilities of all input nodes.

[0215] 7. Based on the replication probability P and its tensor generated by the pointer network, combined with the tensor of the generation probability of all words in the vocabulary output by the multilayer perceptron, predict the generation of the right child node, and use the hidden state of the right child node as the input of the next time step of Right LstmCells.

[0216] 8. Repeat steps 2 to 7. The entire decoding process stops when all leaf nodes of the generated AST are terminal nodes.

[0217] Each predicted left and right child node may continue to generate new left and right child nodes, or may generate a terminal node. If no new nodes need to be calculated, the decoding process stops.

[0218] The target AST is obtained by performing steps (1) to (4) on the Python source code of the example above. The predicted target AST is obtained by... Figures 11a-1 to 11a-3 The structure shown here is composed of, but it should be noted, that to clearly illustrate the predicted target AST, the entire predicted target AST is divided into three parts, consisting of... Figures 11a-1 to 11a-3 Shown separately. Among them, Figure 11a-1 The connecting line a in the middle and Figure 11a-2 The connecting line 'a' in the diagram refers to the same connecting line. Figure 11a-2 The connecting line b in the middle and Figure 11a-3 The connecting line 'b' in the diagram refers to the same connecting line. Figure 11a-2 The connecting line c in the middle and Figure 11a-3 The connecting line 'c' in the diagram refers to the same connecting line. For example... Figures 11a-1 to 11a-3 As shown, circles and ellipses represent nodes on the predicted target AST, with the labeled text being the node name and the number in parentheses below the node name indicating the replication probability.

[0219] The actual target AST is composed of Figures 11b-1 to 11b-3 The structure shown is designed to clearly illustrate the actual target AST. Among them, Figure 11b-1 The connecting line a in the middle and Figure 11b-2 The connecting line 'a' in the diagram refers to the same connecting line. Figure 11b-2 The connecting line b in the middle and Figure 11b-3 The connecting line 'b' in the diagram refers to the same connecting line. Figure 11b-2 The connecting line c in the middle and Figure 11b-3 The connecting line 'c' in the diagram refers to the same connecting line. For example... Figures 11b-1 to 11b-3 As shown, circles and ellipses represent nodes on the actual target AST, with the labeled text being the node name and the number in parentheses below the node name representing the ID value in the dictionary.

[0220] By including Figures 11a-1 to 11a-3 The predicted target AST of the structure shown is related to the inclusion Figures 11b-1 to 11b-3 By comparing the actual target AST with the structure shown, it can be seen that the AST processing method provided by the embodiments of the present invention can accurately and losslessly generate the target AST.

[0221] (5) Output driver code

[0222] The target AST is directly converted into the corresponding driver code in the source code. Direct conversion to code means that each AST is directly converted into the corresponding code, without merging the obtained ASTs into a whole tree as subtrees and then converting them into code.

[0223] Among them, for those containing Figures 11a-1 to 11a-3 The target AST of the structure shown is directly transformed to obtain the prediction code as follows:

[0224] def driver_eoqcu():

[0225] temp_arg_iimztd=get_rand_val_default()

[0226] temp_func_eoqcu=eoqcu(temp_arg_iimztd)

[0227] return True

[0228] Include Figures 11b-1 to 11b-3 The actual driver code (i.e., the actual code) for the target AST of the structure shown is as follows:

[0229] def driver_eoqcu():

[0230] temp_arg_iimztd=get_rand_val_default()

[0231] temp_func_eoqcu=eoqcu(temp_arg_iimztd)

[0232] return True

[0233] In summary, in the programming language processing scenario of code conversion, the embodiments of the present invention convert the source code into an AST and input the AST into a trained neural network model. Through the tree-based embedding layer, encoder, and decoder in the neural network model, relevant processing is performed, which can directly and efficiently convert the AST into the target AST. This achieves direct, end-to-end conversion between the AST and the target AST, and solves the problem that the AST cannot be accurately and losslessly converted into code when OOV exists, thus improving the accuracy of programming language processing.

[0234] Next, taking the programming language processing scenario applied to code conversion as an example, we will provide a detailed introduction to the training method of the neural network model for AST processing provided in this embodiment of the invention.

[0235] Figure 12This is a flowchart illustrating a neural network model training method according to an embodiment of the present invention. See [link / reference]. Figure 12 The neural network model training method of this invention includes the following steps:

[0236] 201. Obtain training samples, which include sample AST and sample target AST.

[0237] The sample AST can be generated by transforming the sample source code, and the sample target AST can be generated by transforming the sample target code, which is generated based on the sample source code. In specific implementations, the sample target code can be generated from the sample source code using fixed rules, or the sample target code can be generated by transforming the sample source code using relevant transformation tools, etc. This embodiment does not specifically limit this.

[0238] Specifically, the process of obtaining the sample AST and the sample target AST can be referred to step 101 in the above embodiment, and will not be repeated here.

[0239] 202. Input the training samples into the preset neural network model, and use the tree-based embedding layer in the neural network model to embed and encode all nodes of the sample AST and all nodes of the target AST of the sample respectively.

[0240] Specifically, a tree-based embedding layer can be used to perform embedding encoding on all nodes of both the sample AST and the target AST using a traversal approach, such as preorder traversal. Here, the tree-based embedding layer performs parameter-trainable embedding encoding on all nodes of the obtained AST, meaning that instead of using a fixed formula for encoding, the network learns custom encoding rules from training to perform embedding encoding.

[0241] An Abstract Syntax Tree (AST) needs to be converted into quantified vectors for analysis and processing in neural network models, typically achieved through spatial embedding. In this embodiment, during the embedding encoding process of the AST input into a tree-based embedding layer, a traversal algorithm is used to control the order of node embedding encoding, ensuring that all nodes of the AST are embedded and encoded according to a tree structure. This allows the embedded AST to distinguish the type of each node and the relationships between nodes. Furthermore, by embedding and encoding all nodes of the AST through a tree-based embedding layer, the spatial structure and syntactic information of the AST can be converted into high-dimensional feature vectors for subsequent calculations.

[0242] 203. The encoded sample AST is processed by the encoder in the neural network model to obtain the encoder-processed sample AST. The encoder is constructed based on a bidirectional neural network.

[0243] The encoder can use a bidirectional tree-based long short-term memory neural network to traverse and update the embedded encoded AST. That is, it first transmits information from leaf nodes to root nodes from bottom to top, and finally gathers the information to the root node of the whole tree. Then, it transmits information from root nodes to leaf nodes from top to bottom.

[0244] 204. The decoder in the neural network model processes the AST sample after the encoder and predicts the nodes of the target AST.

[0245] The decoder can use a tree-based long short-term memory neural network to process the AST processed by the encoder and predict the nodes of the target AST.

[0246] 205. Based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST, a forced-guided training mode is used to train the neural network model.

[0247] The activation of the Teacher Forcing mode is controlled by the relative probability of training progress; that is, the longer the training cycle and the closer the current iteration is to the end of the cycle, the higher the probability of activation. Teacher Forcing is commonly used as a training mode in NLP (Natural Language Processing) tasks such as machine translation, text summarization, and image captioning.

[0248] In the aforementioned neural network model training method, training samples are acquired, including sample ASTs and target ASTs. These training samples are then input into a pre-defined neural network model. The tree-based embedding layer in the neural network model embeds and encodes all nodes of the sample AST, resulting in an embedded and encoded AST. An encoder in the neural network model processes this embedded and encoded AST, producing an encoder-processed AST. The encoder is constructed based on a bidirectional neural network, ensuring that the paths of sibling nodes in the encoder-processed AST are different. This process maintains the structured information of the original tree structure, thus reducing the loss of original information. A decoder in the neural network model processes the encoder-processed AST to predict the nodes of the target AST. Based on the decoder-predicted target AST nodes and the embedded and encoded target AST, a forced-forcing training mode is used to train the neural network model. By introducing this forced-forcing training mode, the network's node prediction process is forced to synchronize with the actual label generation process, i.e., the network is forced to output an AST with the same structure as the label. This allows the network to converge faster and possesses strong structural regularity, which is beneficial for improving the performance of the neural network model.

[0249] In some embodiments, step 202, embedding and encoding all nodes of the sample AST through a tree-based embedding layer in the neural network model, includes:

[0250] When the node name of a node in the sample AST is not the dictionary overflow word OOV, the embedding encoding of the node in the sample AST is obtained based on the ID value of the node in the sample AST.

[0251] When the node name of a node in a sample AST is the dictionary overflow word OOV, the embedding encoding of the node in the sample AST is obtained from the natural distribution of the weight matrix of the embedding layer.

[0252] Specifically, the implementation process of this step can refer to step 102 in the above embodiment, and will not be repeated here.

[0253] In some embodiments, step 203, processing the embedded encoded sample AST using an encoder in a neural network model to obtain the encoder-processed sample AST, may include:

[0254] 2031, the encoder transmits information from leaf nodes to root nodes from bottom to top through the embedded encoded sample AST;

[0255] 2032, the encoder performs bottom-up information transmission from the root node to the leaf node of the sample AST after the bottom-up information transmission is completed, and obtains the encoder-processed sample AST.

[0256] Specifically, the implementation process of this step can refer to steps 1031 to 1032 in the above embodiments, which will not be repeated here.

[0257] In some embodiments, step 2031, which involves transmitting information from leaf nodes to root nodes in the embedded encoded sample AST from bottom to top using an encoder, may include:

[0258] 2031a, by traversing each node of the embedded encoded sample AST through the encoder, the currently traversed node is taken as the current root node;

[0259] 2031b updates the hidden state of the current root node based on the hidden states of its left and right child nodes and the embedding encoding of the current root node, until the feature information of the embedded AST is gathered into the root node of the entire tree, at which point the traversal ends.

[0260] Specifically, the implementation process of steps 2031a to 2031b can refer to steps 1031a to 1031b in the above embodiments, and will not be repeated here.

[0261] In some embodiments, step 2032, where the encoder performs a top-down information transfer from the root node to the leaf node on the sample AST after the bottom-up information transfer is completed, to obtain the encoder-processed sample AST, may include:

[0262] 2032a, by traversing each node of the embedded encoded sample AST through the encoder, the currently traversed node is taken as the current root node;

[0263] 2032b: Based on the hidden state of the parent node of the current root node, the hidden state of the sibling node of the current root node, and the embedding encoding of the current root node, update the hidden state of the current root node until the traversal is completed, and obtain the sample AST after the encoder processing.

[0264] Among them, the sibling nodes of the current root node have the same parent node as the current root node.

[0265] Specifically, the implementation process of steps 2032a to 2032b can refer to steps 1032a to 1032b in the above embodiments, and will not be repeated here.

[0266] In some embodiments, the encoder is constructed based on a deep bidirectional tree-based long short-term memory neural network.

[0267] In some embodiments, step 204 involves processing the encoder-processed sample AST using a decoder in a neural network model to predict the target AST, including:

[0268] Based on the sample AST processed by the encoder, the left and right child nodes of each node in the target AST are predicted by the decoder using an autoregressive decoding method, thus obtaining the target AST.

[0269] In some embodiments, the decoder is constructed based on a tree-based neural network, a pointer network, and a multilayer perceptron.

[0270] In the above steps, based on the sample AST processed by the encoder, the left and right child nodes of each node in the target AST are predicted by the decoder using an autoregressive decoding method to obtain the target AST. This can include:

[0271] 2041, the root node of the sample AST processed by the encoder is used as the initial node of the target AST and input into the tree-based neural network;

[0272] 2042, for the node input into the tree-based neural network at the current time, calculate the hidden states of the left and right child nodes of the node using the tree-based neural network;

[0273] 2043. Based on the AST processed by the encoder and the hidden states of the left and right child nodes of the node, the left and right child nodes of the replica node are predicted by the pointer network to obtain the replication probability and the first prediction result.

[0274] 2044, based on the hidden states of the left and right child nodes of a node, a multilayer perceptron is used to predict and generate the left and right child nodes of the node, thus obtaining the second prediction result;

[0275] 2045, based on the replication probability, the first prediction result and the second prediction result are weighted and merged to generate the left child node and the right child node of the node. The generated left child node and the right child node are respectively used as the input nodes of the tree-based neural network in the next time step.

[0276] 2046. Repeat steps 2042 to 2045 until all leaf nodes of the target AST are generated, and the target AST is obtained.

[0277] In some embodiments, the first prediction result includes a tensor of the probability that all nodes of the sample AST processed by the encoder may be copied; the second prediction result includes a tensor of the generation probability of all words in a preset vocabulary.

[0278] Specifically, the implementation process of steps 2041 to 2046 can refer to steps 1041 to 1046 in the above embodiments, and will not be repeated here.

[0279] In this embodiment, since user-defined words in the language, such as variable names and function names in program code, are inevitably unpredictable, it is impossible to generate them probabilistically. However, probabilistic copying can perfectly solve this problem. By introducing a pointer network into the decoder, the copying probability can be calculated more accurately for all OOVs in the input, solving the problem of information loss caused by the presence of OOVs, which increases the difficulty of convergence and thus helps to obtain more accurate prediction results.

[0280] In some embodiments, the tree-based neural network described above is a tree-based Long Short-Term Memory (LSTM) neural network. This LSTM neural network includes: a left LSTM network for computing the hidden states of the left child nodes of a node, and a right LSTM network for computing the hidden states of the right child nodes of a node. The left and right LSTM networks each contain a set of basic computational units (LSTM Cells). Correspondingly, the decoder includes: a left pointer network and a left multilayer perceptron connected to the left LSTM network, and a right pointer network and a right multilayer perceptron connected to the right LSTM network.

[0281] In some embodiments, step 205 above, training the neural network model using a forced-guided training mode based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST, may include:

[0282] 2051. For each input of the decoder at each time step, the following training operation is performed: the current input of the decoder is replaced by the label node corresponding to the node of the target AST predicted in the previous time step through the correction operation, and then input into the decoder for prediction to obtain the node of the target AST predicted by the decoder at the current time step, where the label node is the node in the embedded encoded sample target AST.

[0283] 2052, repeat the training operation until all nodes of the embedded encoded sample target AST have been used and then stop.

[0284] Specifically, the AST output by the encoder is input to the decoder in a temporal sequence. The focus of TeacherForcing is fixing—during supervised training, because labels exist, the input at time t can be replaced with the output label at time t-1, making it the new input to the decoder. This yields the prediction result at time t, which is then used as the input at time t+1 for a new round of decoding. Similarly, it can be replaced with the output label at time t, and this process is repeated until all labels have been used.

[0285] Figure 13 This is a schematic diagram of the forced guided training process according to an embodiment of the present invention. Figure 13 As shown, the decoder uses a forced-guided training mode to predict the nodes of the AST input at time t. The specific process is as follows:

[0286] 1. Correct the input at time t using fixing, and replace the input at time t with the output label at time t-1.

[0287] 2. Use the output label at time t-1 as the new input to the Decoder.

[0288] 3. Obtain the prediction result at time t.

[0289] 4. Repeat steps 1 through 3 until all tags have been used.

[0290] This invention also provides an AST processing device. Figure 14 This is a schematic diagram of the AST processing device provided in an embodiment of the present invention. See also: Figure 14 As shown, the AST processing device includes:

[0291] Module 301 is used to obtain the Abstract Syntax Tree (AST) to be processed;

[0292] Embedding module 302 is used to input the AST into the trained neural network model, and to embed and encode all nodes of the AST through the tree-based embedding layer in the neural network model to obtain the embedded and encoded AST.

[0293] The encoding module 303 is used to process the embedded encoded AST through the encoder in the neural network model to obtain the encoder-processed AST, wherein the encoder is constructed based on a bidirectional neural network;

[0294] The decoding module 304 is used to process the AST processed by the encoder through the decoder in the neural network model to predict the target AST.

[0295] In some embodiments, the embedding module 302 is specifically used for:

[0296] When the node name of the AST node is not the dictionary overflow word OOV, obtain the embedding code of the AST node based on the ID value of the AST node.

[0297] When the node name of an AST node is a dictionary overflow word OOV, the embedding encoding of the AST node is obtained from the natural distribution of the weight matrix of the embedding layer.

[0298] In some embodiments, the encoding module 303 includes:

[0299] The first encoding unit 3031 is used to transmit information from leaf nodes to root nodes from bottom to top through the encoder after embedding and encoding the AST.

[0300] The second encoding unit 3032 is used to transmit information from the root node to the leaf node from top to bottom in the AST after the bottom-up information transmission is completed by the encoder, so as to obtain the AST after the encoder processing.

[0301] In some embodiments, the first encoding unit 3031 is specifically used for:

[0302] The encoder traverses each node of the embedded encoded AST, and the currently traversed node is taken as the current root node;

[0303] Based on the hidden states of the left and right child nodes of the current root node and the embedding encoding of the current root node, update the hidden state of the current root node until the feature information of the embedded AST is gathered into the root node of the whole tree, and then end the traversal.

[0304] In some embodiments, the second encoding unit 3032 is specifically used for:

[0305] The encoder iterates through each node of the AST after the bottom-up information transmission is completed, and the currently traversed node is taken as the current root node.

[0306] Based on the hidden state of the parent node of the current root node, the hidden state of the sibling nodes of the current root node, and the embedding encoding of the current root node, update the hidden state of the current root node until the traversal is completed, and obtain the AST processed by the encoder.

[0307] Among them, the sibling nodes of the current root node have the same parent node as the current root node.

[0308] In some embodiments, the encoder is constructed from a deep bidirectional tree-based long short-term memory neural network.

[0309] In some embodiments, the decoding module 304 is specifically used for:

[0310] Based on the AST processed by the encoder, the left and right child nodes of each node in the target AST are predicted by the decoder using an autoregressive decoding method, thus obtaining the target AST.

[0311] In some embodiments, the decoder is constructed based on tree-based neural networks, pointer networks, and multilayer perceptrons; the decoding module 304 is specifically used for:

[0312] The root node of the AST processed by the encoder is used as the initial node of the target AST and input into the tree-based neural network.

[0313] For the node currently input into the tree-based neural network, perform the following operations:

[0314] The hidden states of the left and right child nodes of a node are calculated using a tree-based neural network.

[0315] Based on the AST processed by the encoder and the hidden states of the left and right child nodes of the nodes, the left and right child nodes of the replica node are predicted by the pointer network to obtain the replication probability and the first prediction result.

[0316] Based on the hidden states of the left and right child nodes of a node, the left and right child nodes of the generated node are predicted using a multilayer perceptron to obtain a second prediction result.

[0317] Based on the replication probability, the first and second prediction results are weighted and merged to generate the left and right child nodes of the node.

[0318] The left and right child nodes generated by the operation are used as the input nodes of the tree-based neural network in the next time step. The operation is repeated until all leaf nodes of the target AST are generated, and then the target AST is obtained.

[0319] In some embodiments, the first prediction tensor includes a tensor about the probability that all nodes of the AST processed by the encoder may be copied; the second tensor includes a tensor about the generation probability of all words in the preset vocabulary.

[0320] In some embodiments, the tree-based neural network is a tree-based long short-term memory neural network.

[0321] It should be noted that the AST processing device provided in the above embodiments is only illustrated by the division of the above program modules when implementing the AST processing method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the AST processing device can be divided into different program modules to complete all or part of the processing described above. In addition, the AST processing device and the AST processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0322] This invention also provides a neural network model training device. Figure 15 This is a schematic diagram of the neural network model training device according to an embodiment of the present invention. See also... Figure 15 As shown, the neural network model training device includes:

[0323] The acquisition module 401 is used to acquire training samples, which include sample AST and sample target AST.

[0324] The embedding module 402 is used to input training samples into a preset neural network model, and to embed and encode all nodes of the sample AST and all nodes of the sample target AST through the tree-based embedding layer in the neural network model.

[0325] The encoding module 403 is used to process the embedded encoded sample AST through the encoder in the neural network model to obtain the encoder-processed sample AST, wherein the encoder is constructed based on a bidirectional neural network;

[0326] The decoding module 404 is used to process the sample AST processed by the encoder through the decoder in the neural network model and predict the nodes of the target AST.

[0327] Training module 405 is used to train the neural network model using a forced-guided training mode based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST.

[0328] In some embodiments, the embedding module 402 is specifically used for:

[0329] When the node name of a node in the sample AST is not the dictionary overflow word OOV, the embedding encoding of the node in the sample AST is obtained based on the ID value of the node in the sample AST.

[0330] When the node name of a node in a sample AST is the dictionary overflow word OOV, the embedding encoding of the node in the sample AST is obtained from the natural distribution of the weight matrix of the embedding layer.

[0331] In some embodiments, the encoding module 403 includes:

[0332] The first encoding unit 4031 is used to transmit information from leaf nodes to root nodes from bottom to top through the encoder on the embedded encoded sample AST.

[0333] The second encoding unit 4032 is used to perform root node to leaf node information transmission from top to bottom on the sample AST after the bottom-up information transmission is completed by the encoder, so as to obtain the sample AST after encoder processing.

[0334] In some embodiments, the first encoding unit 4031 is specifically used for:

[0335] The encoder traverses each node of the embedded encoded sample AST, and the currently traversed node is taken as the current root node;

[0336] Based on the hidden states of the left and right child nodes of the current root node and the embedding encoding of the current root node, update the hidden state of the current root node until the feature information of the embedded AST is gathered into the root node of the whole tree, and then end the traversal.

[0337] In some embodiments, the second encoding unit 4032 is specifically used for:

[0338] The encoder iterates through each node of the sample AST after the bottom-up information transmission is completed, and the currently traversed node is taken as the current root node.

[0339] Based on the hidden state of the parent node of the current root node, the hidden state of the sibling nodes of the current root node, and the embedding encoding of the current root node, update the hidden state of the current root node until the traversal is completed, and obtain the sample AST after the encoder processing.

[0340] Among them, the sibling nodes of the current root node have the same parent node as the current root node.

[0341] In some embodiments, the encoder is constructed based on a deep bidirectional tree-based long short-term memory neural network.

[0342] In some embodiments, the decoding module 404 is specifically used for:

[0343] Based on the sample AST processed by the encoder, the left and right child nodes of each node in the target AST are predicted by the decoder using an autoregressive decoding method, thus obtaining the target AST.

[0344] In some embodiments, the decoder is constructed based on a tree-based neural network, a pointer network, and a multilayer perceptron; the decoding module 1304 is specifically used for:

[0345] The root node of the sample AST processed by the encoder is used as the initial node of the target AST and input into the tree-based neural network.

[0346] For the node currently input into the tree-based neural network, perform the following operations:

[0347] The hidden states of the left and right child nodes of a node are calculated using a tree-based neural network.

[0348] Based on the AST of the sample processed by the encoder and the hidden states of the left and right child nodes of the node, the left and right child nodes of the replica node are predicted by the pointer network to obtain the replication probability and the first prediction result.

[0349] Based on the hidden states of the left and right child nodes of a node, the left and right child nodes of the generated node are predicted using a multilayer perceptron to obtain a second prediction result.

[0350] Based on the replication probability, the first and second prediction results are weighted and merged to generate the left and right child nodes of the node.

[0351] The left and right child nodes generated by the operation are used as the input nodes of the tree-based neural network in the next time step. The operation is repeated until all leaf nodes of the target AST are generated, and then the target AST is obtained.

[0352] In some embodiments, the first tensor includes a tensor relating the probability that all nodes of the sample AST processed by the encoder may be copied; the second tensor includes a tensor relating the generation probability of all words in a preset vocabulary.

[0353] In some embodiments, the tree-based neural network is a tree-based long short-term memory neural network.

[0354] In some embodiments, the training module 405 is specifically used for:

[0355] For each time step of the decoder input, perform the following training operation:

[0356] The current input of the decoder is replaced by the label node corresponding to the node of the target AST predicted in the previous time step through the correction operation, and then input into the decoder for prediction to obtain the node of the target AST predicted by the decoder at the current time step. Here, the label node is the node in the target AST of the embedded encoded sample.

[0357] Repeat the training operation until all nodes of the embedded encoded sample target AST have been used, then stop.

[0358] It should be noted that the neural network model training device provided in the above embodiments is only illustrated by the division of the above program modules when implementing the neural network model training method. In practical applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the neural network model training device can be divided into different program modules to complete all or part of the processing described above. In addition, the neural network model training device and the embodiment of the neural network model training method provided in the above embodiments belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0359] Figure 16 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention; as shown below. Figure 16 As shown, the computer device 500 includes: a processor 501 and a memory 502 for storing computer programs capable of running on the processor; wherein,

[0360] When a computer device is used in an AST processing method, the processor 501 is used to run a computer program to implement the steps of any of the above embodiments of the AST processing method.

[0361] When a computer device is used in a neural network model training method, and the processor 501 is used to run a computer program, the steps of any of the neural network model training methods described in the above embodiments are implemented.

[0362] In practical applications, computer device 500 may also include at least one network interface 503. The various components in computer device 500 are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 16 All buses are labeled as bus system 504. The number of processors 501 can be at least one. Network interface 503 is used for wired or wireless communication between computer device 500 and other devices.

[0363] The memory 502 in this embodiment of the invention is used to store various types of data to support the operation of the computer device 500.

[0364] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by processor 501. Processor 501 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 501 or by instructions in software form. The processor 501 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 501 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of the present invention can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 502. Processor 501 reads the information in memory 502 and combines its hardware to complete the steps of the aforementioned method.

[0365] In an exemplary embodiment, the computer device 500 may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers (MCUs), microprocessors, or other electronic components to perform the aforementioned method.

[0366] This invention also provides a computer-readable storage medium storing a computer program thereon; when the computer-readable storage medium is applied to the AST processing method, the computer program is executed by a processor to implement the steps of any of the above-described AST processing methods.

[0367] When a computer-readable storage medium is used in a neural network model training method, and the computer program is executed by a processor, the steps of the neural network model training method of any of the above embodiments are implemented.

[0368] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0369] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the various embodiments of the present invention, all functional units can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0370] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0371] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0372] It should be noted that terms such as "first" and "second" are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0373] Furthermore, the technical solutions described in the embodiments of the present invention can be combined arbitrarily without conflict.

[0374] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An AST processing method, characterized in that, The method includes: Obtain the Abstract Syntax Tree (AST) to be processed; The AST is input into a trained neural network model, and all nodes of the AST are embedded and encoded through a tree-based embedding layer in the neural network model to obtain the embedded and encoded AST. The AST after embedding is processed by the encoder in the neural network model to obtain the AST after encoder processing, wherein the encoder is constructed based on a bidirectional neural network; The target AST is predicted by processing the AST processed by the encoder through the decoder in the neural network model, wherein the decoder is constructed based on a tree-based neural network, a pointer network, and a multilayer perceptron. The step of processing the AST processed by the encoder through the decoder in the neural network model to predict the target AST includes: The root node of the AST processed by the encoder is used as the initial node of the target AST and input into the tree-based neural network. For the node currently input into the tree-based neural network, perform the following operations: The hidden states of the left and right child nodes of the node are calculated using the tree-based neural network. Based on the AST processed by the encoder and the hidden states of the left and right child nodes of the node, the left and right child nodes of the node are predicted and copied through the pointer network to obtain the copying probability and the first prediction result. Based on the hidden states of the left and right child nodes of the node, the left and right child nodes of the node are predicted and generated by the multilayer perceptron to obtain a second prediction result. Based on the replication probability, the first prediction result and the second prediction result are weighted and merged to generate the left child node and right child node of the node; The left and right child nodes generated by the operation are used as the nodes of the tree-based neural network in the next time step. The operation is repeated until all leaf nodes of the target AST are generated, and the target AST is obtained.

2. The method according to claim 1, characterized in that, The step of embedding and encoding all nodes of the AST through a tree-based embedding layer in the neural network model to obtain the embedded and encoded AST includes: When the node name of the node in the AST is not the dictionary overflow word OOV, the embedding encoding of the node in the AST is obtained according to the ID value of the node in the AST. When the node name of the node in the AST is the dictionary overflow word OOV, the embedding code of the node in the AST is obtained from the natural distribution of the weight matrix of the embedding layer.

3. The method according to claim 1 or 2, characterized in that, The step of processing the embedded AST using the encoder in the neural network model to obtain the encoder-processed AST includes: The encoder transmits information from leaf nodes to root nodes in the embedded AST from bottom to top. The encoder performs bottom-up information transmission on the AST, then performs top-down information transmission from the root node to the leaf node, to obtain the encoder-processed AST.

4. The method according to claim 3, characterized in that, The step of transmitting information from leaf nodes to the root node in the AST after embedding and encoding by the encoder from bottom to top includes: The encoder traverses each node of the embedded encoded AST, and the currently traversed node is taken as the current root node; The hidden state of the current root node is updated based on the hidden states of its left and right child nodes and the embedding encoding of the current root node. The traversal ends when the feature information of the AST after embedding encoding is gathered into the root node of the entire tree.

5. The method according to claim 3, characterized in that, The process of transmitting information from the root node to the leaf node of the AST after bottom-up information transmission via the encoder to obtain the encoder-processed AST includes: The encoder traverses each node of the AST after the bottom-up information transmission is completed, and the currently traversed node is taken as the current root node. Based on the hidden state of the parent node of the current root node, the hidden state of the sibling node of the current root node, and the embedding encoding of the current root node, update the hidden state of the current root node until the traversal is completed, and obtain the AST processed by the encoder. Wherein, the sibling nodes of the current root node have the same parent node as the current root node.

6. The method according to claim 1 or 2, characterized in that, The encoder is constructed based on a deep bidirectional tree-structured long short-term memory neural network.

7. The method according to claim 1, characterized in that, The first prediction result includes a tensor of the probability that all nodes of the AST processed by the encoder may be copied; the second prediction result includes a tensor of the generation probability of all words in the preset vocabulary.

8. The method according to claim 1 or 7, characterized in that, The tree-based neural network is a tree-based long short-term memory neural network.

9. The method according to claim 1 or 2, characterized in that, When the AST is the AST of the source code, the target AST is used to transform and generate the target code.

10. A method for training a neural network model, characterized in that, The method includes: Obtain training samples, which include sample AST and sample target AST; The training samples are input into a preset neural network model, and the tree-based embedding layer in the neural network model is used to embed and encode all nodes of the sample AST and all nodes of the sample target AST respectively. The encoded sample AST is processed by the encoder in the neural network model to obtain the encoder-processed sample AST, wherein the encoder is constructed based on a bidirectional neural network; The decoder in the neural network model processes the AST sample after the encoder and predicts the nodes of the target AST. Based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST, the neural network model is trained using a forced-guided training mode; wherein, the decoder is constructed based on tree-based neural networks, pointer networks, and multilayer perceptrons; The step of processing the sample AST processed by the encoder through the decoder in the neural network model to predict the target AST includes: The root node of the sample AST processed by the encoder is used as the initial node of the target AST and input into the tree-based neural network. For the node currently input into the tree-based neural network, perform the following operations: The hidden states of the left and right child nodes of the node are calculated using the tree-based neural network. Based on the sample AST processed by the encoder and the hidden states of the left and right child nodes of the node, the left and right child nodes of the node are predicted and copied through the pointer network to obtain the copying probability and the first prediction result. Based on the hidden states of the left and right child nodes of the node, the left and right child nodes of the node are predicted and generated by the multilayer perceptron to obtain a second prediction result. Based on the replication probability, the first prediction result and the second prediction result are weighted and merged to generate the left child node and right child node of the node; The left and right child nodes generated by the operation are used as the nodes of the tree-based neural network in the next time step. The operation is repeated until all leaf nodes of the target AST are generated, and the target AST is obtained.

11. The method according to claim 10, characterized in that, The embedding encoding of all nodes of the sample AST through the tree-based embedding layer in the neural network model includes: When the node name of the node in the sample AST is not the dictionary overflow word OOV, the embedding code of the node in the sample AST is obtained according to the ID value of the node in the sample AST. When the node name of a node in the sample AST is the dictionary overflow word OOV, the embedding code of the node in the sample AST is obtained from the natural distribution of the weight matrix of the embedding layer.

12. The method according to claim 10 or 11, characterized in that, The step of processing the embedded AST of the sample through the encoder in the neural network model to obtain the encoder-processed sample AST includes: The encoder performs bottom-up information transfer from leaf nodes to root nodes on the embedded AST sample; The encoder performs bottom-up information transmission on the sample AST, then performs top-down information transmission from the root node to the leaf node, to obtain the encoder-processed sample AST.

13. The method according to claim 12, characterized in that, The step of transmitting information from leaf nodes to root nodes in the embedded AST of the sample through the encoder from bottom to top includes: The encoder traverses each node of the embedded encoded sample AST, and the currently traversed node is taken as the current root node; The hidden state of the current root node is updated based on the hidden states of its left and right child nodes and the embedding encoding of the current root node. The traversal ends when the feature information of the sample AST after embedding encoding is gathered into the root node of the entire tree.

14. The method according to claim 12, characterized in that, The process of passing information from the root node to the leaf node of the sample AST after bottom-up information transmission via the encoder to obtain the encoder-processed sample AST includes: The encoder traverses each node of the sample AST after the bottom-up information transmission is completed, and the currently traversed node is taken as the current root node. Based on the hidden state of the parent node of the current root node, the hidden state of the sibling node of the current root node, and the embedding encoding of the current root node, update the hidden state of the current root node until the traversal is completed, and obtain the sample AST processed by the encoder. Wherein, the sibling nodes of the current root node have the same parent node as the current root node.

15. The method according to claim 10 or 11, characterized in that, The encoder is constructed based on a deep bidirectional tree-structured long short-term memory neural network.

16. The method according to claim 10, characterized in that, The first prediction result includes a tensor of the probability that all nodes of the sample AST processed by the encoder may be copied; the second prediction result includes a tensor of the generation probability of all words in the preset vocabulary.

17. The method according to claim 10 or 16, characterized in that, The tree-based neural network is a tree-based long short-term memory neural network.

18. The method according to claim 10 or 11, characterized in that, The neural network model is trained using a forced-guided training mode, based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST, including: For each time step of the decoder's input, the following training operation is performed: The current input of the decoder is replaced by the label node corresponding to the node of the target AST predicted in the previous time step through a correction operation, and then input into the decoder for prediction to obtain the node of the target AST predicted by the decoder at the current time step, wherein the label node is the node in the target AST of the embedded encoding sample. Repeat the training operation until all nodes of the embedded encoded target AST of the sample are used, then stop.

19. The method according to claim 10 or 11, characterized in that, When the sample AST is the AST of the sample source code, the sample target AST is the AST of the sample target code obtained by converting the sample source code.

20. An AST processing device, characterized in that, The device includes: The acquisition module is used to acquire the Abstract Syntax Tree (AST) to be processed. An embedding module is used to input the AST into a trained neural network model, and to embed and encode all nodes of the AST through a tree-based embedding layer in the neural network model to obtain the embedded and encoded AST. An encoding module is used to process the embedded encoded AST through the encoder in the neural network model to obtain the encoder-processed AST, wherein the encoder is constructed based on a bidirectional neural network; The decoding module is used to process the AST processed by the encoder through the decoder in the neural network model to predict the target AST; The decoder is constructed based on tree-based neural networks, pointer networks, and multilayer perceptrons. The decoding module is further configured to input the root node of the AST processed by the encoder as the initial node of the target AST into the tree-based neural network; for the node input into the tree-based neural network at the current time, perform the following operations: calculate the hidden states of the left and right child nodes of the node through the tree-based neural network; based on the AST processed by the encoder and the hidden states of the left and right child nodes of the node, predict and copy the left and right child nodes of the node through the pointer network to obtain the copying probability and a first prediction result; based on the hidden states of the left and right child nodes of the node, predict and generate the left and right child nodes of the node through the multilayer perceptron to obtain a second prediction result; based on the copying probability, weight and merge the first prediction result and the second prediction result to generate the left and right child nodes of the node; use the left and right child nodes generated by the operation as the nodes input into the tree-based neural network at the next time, and repeat the operation until all leaf nodes of the target AST are generated, thus obtaining the target AST.

21. A neural network model training device, characterized in that, The device includes: The acquisition module is used to acquire training samples, which include sample AST and sample target AST; An embedding module is used to input the training samples into a preset neural network model, and to embed and encode all nodes of the sample AST and all nodes of the sample target AST through a tree-based embedding layer in the neural network model. An encoding module is used to process the embedded encoded sample AST through the encoder in the neural network model to obtain the encoder-processed sample AST, wherein the encoder is constructed based on a bidirectional neural network; The decoding module is used to process the sample AST processed by the encoder through the decoder in the neural network model to predict the nodes of the target AST; The training module is used to train the neural network model using a forced-guided training mode based on the nodes of the target AST predicted by the decoder and the embedded encoded sample target AST. The decoder is constructed based on tree-based neural networks, pointer networks, and multilayer perceptrons. The decoding module is further configured to input the root node of the sample AST processed by the encoder as the initial node of the target AST into the tree-based neural network; for the node input into the tree-based neural network at the current time, perform the following operations: calculate the hidden states of the left and right child nodes of the node through the tree-based neural network; based on the sample AST processed by the encoder and the hidden states of the left and right child nodes of the node, predict and copy the left and right child nodes of the node through the pointer network to obtain the copying probability and a first prediction result; based on the hidden states of the left and right child nodes of the node, predict and generate the left and right child nodes of the node through the multilayer perceptron to obtain a second prediction result; based on the copying probability, weight and merge the first prediction result and the second prediction result to generate the left and right child nodes of the node; use the left and right child nodes generated by the operation as the nodes input into the tree-based neural network at the next time, and repeat the operation until all leaf nodes of the target AST are generated, thus obtaining the target AST.

22. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the AST processing method according to any one of claims 1 to 9; or, when the processor executes the program, it implements the steps of the neural network model training method according to any one of claims 10 to 19.

23. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the AST processing method according to any one of claims 1 to 9; or, when the computer program is executed by a processor, it implements the steps of the neural network model training method according to any one of claims 10 to 19.

Images