Input encoding method and system for pre-trained networks based on code hierarchy

Through pre-training network based on code hierarchy, word segmentation and token vocabulary tables, obtain code sequence embedding and hierarchical value embedding, and fuse position encoding, the problem of high training complexity and time-consuming in traditional methods is solved, and more efficient code understanding and analysis.

CN118363657BActive Publication Date: 2024-11-15HEFEI UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410568432.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-09
Publication Date
2024-11-15
Estimated Expiration
2044-05-09

AI Technical Summary

Technical Problem

Traditional code processing methods are complex and time-consuming to train, making it difficult to effectively capture the structure and semantic information of the code.

Method used

Through a pre-trained network based on the code hierarchy, the token vocabulary is constructed by word segmentation, the code sequence embedding and hierarchical value embedding are obtained, the position encoding is integrated, and the input vector of the pre-trained network is generated.

Benefits of technology

It improves the accuracy and efficiency of code understanding and analysis, can better capture the structure and semantic information of the code, and reduces the complexity and time-consuming encoding.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118363657B_ABST
    Figure CN118363657B_ABST
Patent Text Reader

Abstract

The present invention provides an input encoding method, system, storage medium and electronic device of a pre-trained network based on a code hierarchy structure, and relates to the field of machine learning. In the present invention, the original code sequence is segmented and a corresponding token word list is constructed; the code sequence after segmentation is transformed using the token word list to obtain a code sequence embedding; according to the indentation level of the original code sequence, a hierarchical value sequence embedding is obtained; according to the hierarchical value sequence embedding, a hierarchical encoding result is obtained; and the hierarchical value sequence embedding of each row is spliced ​​with the code sequence embedding to which the row belongs to obtain a vector representation result; the position encoding result of the original code sequence is obtained; the vector representation result, the hierarchical encoding result and the position encoding result are fused to obtain the input vector of the pre-trained network. By using the code hierarchy structure, compared with the prior art, the structure and semantic information of the code can be better captured, the accuracy and effect of encoding can be improved, and the efficiency and speed of encoding can be improved at the same time.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of machine learning, and in particular to an input encoding method, system, storage medium and electronic device for a pre-trained network based on a code hierarchy. Background Art

[0002] With the rapid development of deep learning, code understanding and analysis has always been a challenging task. As the scale and complexity of software increase, the need for automated processing and understanding of code has become more and more urgent. To this end, researchers have proposed various technologies and methods to solve these problems.

[0003] However, traditional code processing methods are mainly based on lexical and syntactic analysis, parsing the code into a series of tokens or syntax tree nodes, and then using this information for subsequent analysis and processing, which has limited understanding and analysis of the code; some current mainstream processing directions, such as using the structure of the AST abstract syntax tree to supplement structural information, have problems such as high training complexity and long time consumption. Summary of the invention

[0004] 1. Technical issues to be solved

[0005] In view of the deficiencies in the prior art, the present invention provides an input encoding method, system, storage medium and electronic device for a pre-trained network based on a code hierarchy, which solves the technical problems of high training complexity and long training time.

[0006] (II) Technical solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0008] A method for encoding input of a pre-trained network based on a code hierarchy, comprising:

[0009] Segment the original code sequence and construct the corresponding token vocabulary;

[0010] Using the token vocabulary to transform the code sequence after word segmentation to obtain the code sequence embedding;

[0011] According to the indentation level of the original code sequence, obtaining a level value sequence embedding;

[0012] Obtaining a hierarchical coding result according to the hierarchical value sequence embedding; and concatenating the hierarchical value sequence embedding of each row with the code sequence embedding to which the row belongs to obtain a vector representation result;

[0013] Obtaining a position encoding result of the original code sequence;

[0014] The vector representation result, the hierarchical encoding result and the position encoding result are fused to obtain an input vector of the pre-trained network.

[0015] Preferably, the process of segmenting the original code sequence and constructing a corresponding vocabulary includes:

[0016] Selecting a syntax analyzer corresponding to the programming language used by the original code sequence according to the language type contained in the data sample;

[0017] Load the local parser and the corresponding language;

[0018] The original code sequence is segmented by the loaded syntax analyzer, and a corresponding token vocabulary is constructed; each token group contains a number of sub-tokens.

[0019] Preferably, the method of converting the segmented code sequence using the token vocabulary to obtain the code sequence embedding includes:

[0020] The token word list is used to replace the code sequence after word segmentation, and the token is filled so that each token group contains k sub-tokens to obtain the code sequence S token ;

[0021] For the code sequence S token Processing is performed to obtain the code sequence embedded in E token ;

[0022] E tokeni,j =Embedding(token i,j ,n token ,d)

[0023] Among them, token i,j Represents the code sequence S token The jth subtoken in the i-th token, n token It represents the range of token values ​​contained in the token vocabulary, Embedding represents the code embedding operation, and d represents the embedding dimension.

[0024] Preferably, the step of acquiring the level value sequence embedding according to the indentation level of the original code sequence comprises:

[0025] The original code sequence is formatted normally and traversed line by line, and the indentation of each line is used as the level value to obtain the code sequence s token The corresponding level value sequence S level ;

[0026] For the level value sequence S leve deal with:

[0027] E level =Embedding′(S level ,d)

[0028] Among them, E level represents hierarchical value sequence embedding, and Embedding′ represents the hierarchical embedding operation.

[0029] Preferably, the hierarchical value sequence is embedded and normalized to obtain the hierarchical coding result, which is expressed as:

[0030] E′ level =Sigmoid(E level )

[0031] Among them, E′ level Represents the hierarchical encoding result, Sigmoid represents the nonlinear activation function;

[0032] The hierarchical value sequence embedding of each row is concatenated with the code sequence embedding to which the row belongs to obtain a vector representation result; it is expressed as:

[0033] E′ token =inear(concat(E level ,E token ,0),k*d,d)

[0034] Among them, E′ token Indicates that the vector represents the result; Linear indicates a linear transformation operation; concat indicates a concatenation operation.

[0035] Preferably, the cosine position encoding method is used to obtain the position encoding result PE of the original code sequence.

[0036] Preferably, the fusion of the vector representation result, the hierarchical encoding result and the position encoding result to obtain the input vector of the pre-trained network is expressed as:

[0037] input=W h · l ' evel + p ·E+E′ token

[0038] Among them, input represents the input vector, W h , W p Represent the corresponding weights respectively.

[0039] An input encoding system for a pre-trained network based on a code hierarchy, comprising:

[0040] The word segmentation module is used to segment the original code sequence and construct the corresponding token vocabulary;

[0041] A conversion module, used to convert the code sequence after word segmentation using the token vocabulary to obtain the code sequence embedding;

[0042] An acquisition module, used for acquiring a level value sequence embedding according to the indentation level of the original code sequence;

[0043] A first encoding module is used to obtain a hierarchical encoding result according to the hierarchical value sequence embedding; and to concatenate the hierarchical value sequence embedding of each row with the code sequence embedding to which the row belongs to obtain a vector representation result;

[0044] A second encoding module, used to obtain the position encoding result of the original code sequence;

[0045] A fusion module is used to fuse the vector representation result, the hierarchical encoding result and the position encoding result to obtain an input vector of the pre-trained network.

[0046] A storage medium stores a computer program for input encoding of a pre-trained network based on a code hierarchy, wherein the computer program enables a computer to execute the input encoding method as described above.

[0047] An electronic device, comprising:

[0048] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the input encoding method as described above.

[0049] (III) Beneficial effects

[0050] The present invention provides an input encoding method, system, storage medium and electronic device for a pre-trained network based on a code hierarchy. Compared with the prior art, it has the following beneficial effects:

[0051] In the present invention, the original code sequence is segmented and the corresponding token word list is constructed; the token word list is used to transform the segmented code sequence to obtain the code sequence embedding; according to the indentation level of the original code sequence, the level value sequence embedding is obtained; according to the level value sequence embedding, the level coding result is obtained; and the level value sequence embedding of each line is spliced ​​with the code sequence embedding to which the line belongs to obtain the vector representation result; the position coding result of the original code sequence is obtained; the vector representation result, the level coding result and the position coding result are fused to obtain the input vector of the pre-trained network. By using the code hierarchy, compared with the prior art, the structure and semantic information of the code can be better captured, the accuracy and effect of coding can be improved, and the efficiency and speed of coding can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 A block diagram of an input encoding method for a pre-trained network based on a code hierarchy provided in an embodiment of the present invention;

[0054] Figure 2 A flowchart of an input encoding method for a pre-trained network based on a code hierarchy provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] The embodiments of the present application solve the technical problems of high training complexity and long training time by providing an input encoding method, system, storage medium and electronic device for a pre-trained network based on a code hierarchy.

[0057] The technical solution in the embodiment of the present application is to solve the above technical problems, and the overall idea is as follows:

[0058] The encoding method provided by the embodiment of the present invention is applicable to multiple fields, such as code summarization, code translation, etc. In these fields, the encoding method of the present invention can improve the performance and effect of the model and better meet actual needs.

[0059] Specifically: the original code sequence is segmented and the corresponding token vocabulary is constructed; the token vocabulary is used to transform the segmented code sequence to obtain the code sequence embedding; according to the indentation level of the original code sequence, the hierarchical value sequence embedding is obtained; according to the hierarchical value sequence embedding, the hierarchical encoding result is obtained; and the hierarchical value sequence embedding and the code sequence embedding are concatenated to obtain the vector representation result; the position encoding result of the original code sequence is obtained; the vector representation result, the hierarchical encoding result and the position encoding result are fused to obtain the input vector of the pre-trained network.

[0060] By using the code hierarchy, compared with the existing technology, the structure and semantic information of the code can be better captured, the accuracy and effect of coding can be improved, and the efficiency and speed of coding can be improved.

[0061] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0062] Embodiment 1:

[0063] like Figure 1 As shown, an embodiment of the present invention provides an input encoding method for a pre-trained network based on a code hierarchy, comprising:

[0064] S1. Segment the original code sequence and construct the corresponding token vocabulary;

[0065] S2. Using the token vocabulary to transform the code sequence after word segmentation, and obtain the code sequence embedding;

[0066] S3, acquiring a level value sequence embedding according to the indentation level of the original code sequence;

[0067] S4, acquiring a hierarchical coding result according to the hierarchical value sequence embedding; and concatenating the hierarchical value sequence embedding of each row with the code sequence embedding to which the row belongs, to acquire a vector representation result;

[0068] S5, obtaining the position encoding result of the original code sequence;

[0069] S6. Fusing the vector representation result, the hierarchical encoding result and the position encoding result to obtain an input vector of the pre-trained network.

[0070] Compared with the traditional method using natural language methods, the input encoding method based on the pre-trained network of the code hierarchy provided by the embodiment of the present invention can better capture the hierarchy and semantic information of the code, and improve the accuracy and effect of code understanding and analysis.

[0071] Compared with the traditional AST-based processing method which usually only focuses on the relative relationship between nodes, the code hierarchy-based structure provided by the embodiment of the present invention can supplement the global hierarchical information and help the model better understand the structure.

[0072] In addition, by using pre-trained neural network models (i.e., the pre-trained networks mentioned above), we can utilize the rich information in large-scale code bases to improve the generalization ability and effect of the model. Moreover, the model's processing of structures can theoretically support all languages ​​and tasks, and after fine-tuning, it can be applied to various code-related tasks, such as code search, code recommendation, and code auto-completion.

[0073] Next, we will combine Figure 2 The following are the steps of the above solution:

[0074] In step S1, the original code sequence is segmented and the corresponding token vocabulary is constructed; including:

[0075] S11, selecting a syntax analyzer corresponding to the programming language used by the original code sequence according to the language type contained in the data sample;

[0076] S12, loading the local parser and the corresponding language;

[0077] S13. Segment the original code sequence by the loaded syntax analyzer and construct a corresponding token vocabulary; wherein each token group contains a number of sub-tokens.

[0078] See also Figure 2 When executing, first obtain a data set that may contain code sequences in multiple languages, classify them according to the language type of each data set (Guesslang can be used to judge the code language in subsequent reasoning), and then use the corresponding language list to create a syntax analyzer. Here, Pygments or NLTK can be used to create a syntax analyzer.

[0079] Input the original code sequence in the training data set into the syntax analyzer to obtain the corresponding word segmentation result T = (S, Voc_t), where Voc_t is the token vocabulary, S is the sequence of the corresponding code segment after word segmentation, and each token group in the sequence may contain multiple sub-tokens, which can be regarded as a feature of the token itself.

[0080] In step S2, the token word list is used to transform the code sequence after word segmentation to obtain the code sequence embedding; including:

[0081] S21, using the token vocabulary to replace the code sequence after word segmentation, and fill the token so that each token group contains k sub-tokens, to obtain the code sequence S token ;

[0082] S22, the code sequence S token Processing is performed to obtain the code sequence embedded in E token ;

[0083] E tokeni,j =Embedding(token i,j ,n token ,d)

[0084] Among them, token i,j Represents the code sequence S token The jth subtoken in the i-th token, n token It represents the range of token values ​​contained in the token vocabulary, Embedding represents the code embedding operation, and d represents the embedding dimension.

[0085] During the specific execution, the token is replaced according to the vocabulary Voc_t, the sub-tokens contained in the token group in S are converted into corresponding value representations, and filled with special token values ​​that do not contain semantics, that is, the code sequence S is processed into an n*l sequence, n is the number of sub-tokens contained in each token group after filling, and l is the length of the code sequence.

[0086] In step S3, according to the indentation level of the original code sequence, a level value sequence embedding is obtained; including

[0087] S31, format the original code sequence normally and traverse it line by line, take the indentation of each line as the level value, and obtain the code sequence S tokwn The corresponding level value sequence S levwl ;

[0088] During the specific execution, the original code sequence is formatted normally and traversed line by line, and the indentation of each line is obtained as the level value level. The level of each line is mapped to each token group in the line and processed into a sequence, that is, S is obtained. level is a sequence of length l.

[0089] S32, the level value sequence S level deal with:

[0090] E level =Embedding′(S level ,d)

[0091] Among them, E level represents hierarchical value sequence embedding, and Embedding′ represents the hierarchical embedding operation.

[0092] In step S4, a hierarchical coding result is obtained according to the hierarchical value sequence embedding; and the hierarchical value sequence embedding of each row is concatenated with the code sequence embedding to which the row belongs to obtain a vector representation result.

[0093] The hierarchical value sequence is embedded and normalized to obtain the hierarchical coding result, which is expressed as:

[0094] E′ level =Sigmoid(E level )

[0095] Among them, E′ level Represents the hierarchical encoding result, and Sigmoid represents the nonlinear activation function.

[0096] And concatenate the hierarchical value sequence embedding of each row with the code sequence embedding to which the row belongs to obtain a vector representation result; expressed as:

[0097] E′ token =Lonear(concat(E level ,E token ,0),k*d,d)

[0098] Among them, E′ token Indicates that the vector represents the result; Linear indicates a linear transformation operation; concat indicates a concatenation operation.

[0099] In step S5, the position encoding result of the original code sequence is obtained.

[0100] Exemplarily, the embodiment of the present invention adopts cosine position coding to obtain the position coding result PE of the original code sequence; the calculation method is as follows:

[0101] PE (pos,2i) =sin(pos / 10000 2i / d )

[0102] PE (pos,2i+1) =cos(pos / 10000 2i / d )

[0103] Among them, pos represents the position of the token in the sequence, 2i represents odd dimensions, and 2i+1 represents even dimensions.

[0104] In step S6, the vector representation result, the hierarchical encoding result and the position encoding result are fused to obtain an input vector of the pre-trained network.

[0105] The vector representation result E′ obtained in the fusion step S4 token , the hierarchical coding result E′ obtained in step S4 level , and the position encoding result PE obtained in step S5, to obtain the input vector of the pre-trained network so that it can be input into the encoder for training later.

[0106] The fusion process is expressed as:

[0107] input=W h ·E′ level +W p PE+E′ token

[0108] Among them, input represents the input vector, W h , W p Represent the corresponding weights respectively.

[0109] To demonstrate the superiority of this application, the following is a detailed description of a Transformer structure for a code summarization task:

[0110] For code summarization tasks, the training data pairs are generally the original code that has been classified and the code summary in the form of comments. When training the model, the data is read from the training set and divided into source (source sequence) and target (target sequence). The source sequence as input is embedded and encoded and input into the model.

[0111] The processing of the source part first requires reading the data set and extracting the code segments, filtering the extracted code segments according to the set threshold, and discarding the code segments whose length exceeds the range. Then, the corresponding syntax analyzer is obtained according to the language classification of the code, and the syntax analyzer is used to segment the code to obtain the corresponding segmentation result T = (S, Voc_t), where Voc_t is the token vocabulary, S is the segmented sequence of the corresponding code segment, and each token group in the sequence may contain multiple sub-tokens, that is, the structure of each token group is

[0112] For the obtained code sequence, replace the token according to the vocabulary Voc_t, convert the sub-tokens contained in the token group in S into corresponding value representations, and use special token values ​​that do not contain semantics to fill them, that is, process the code sequence S into an n*l sequence, where l is the length of the code sequence and n is the number of sub-tokens contained in each token group after filling, which can be adjusted according to the actual token situation. According to the calculation formula Get the word embedding representation of the code sequence.

[0113] Since there may be blank lines in the code data, or the indentation may not be strict in code languages ​​other than Python, the original code segment is formatted normally to obtain the standard code format.

[0114] Traverse the original code sequence line by line, get the indentation of each line as the level value, map the level of each line to each token group in the line, and process them into a sequence, that is, get a sequence S of length l level , and the sequence embedding representation E is also obtained level =Embedding(S level ,d). The hierarchical embedding representation is used as a feature dimension of the code and the word embedding representation of the code sequence is concatenated in the feature dimension of the token to obtain the (n+1)*l*d code embedding matrix E token .

[0115] Transformed into code representation E′ through a linear layer token , the calculation formula is E′ token =Linear(E token ,k*d,d), where k represents the number of sub-tokens in each token group and d represents the embedding dimension

[0116] For E level Normalized processing is performed according to the calculation formula: E′ level =Sigmoid(E level ) to obtain the hierarchical coding result E′ level , where sigmoid is a specific nonlinear activation function; the cosine position encoding method is used to calculate the position encoding of the input code, and the calculation method is as follows:

[0117] PE (pos,2i) =sin(pos / 10000 2i / d )

[0118] PE (pos,2i+1) =cos(pos / 10000 2i / )

[0119] Positional encoding hierarchical embedding and positional encoding only need to be directly weighted summed from front to back with the vector representation of the code sequence, and the calculation input = W h · level + p E+Se, that is, the encoder input vector is obtained.

[0120] The input vector obtained will be input into the Transformer encoder, which consists of multiple identical layers, each of which contains a self-attention layer and a feedforward neural network layer. The output of each layer will be used as the input of the next layer until the output of the last layer is input into the decoder structure. The decoder also consists of multiple identical layers, each of which contains a self-attention layer, an encoder-decoder attention layer, and a feedforward neural network layer, and finally obtains the model's predicted value.

[0121] The model target value is the vector representation of the target part contained in the dataset, that is, the code summary in the form of comments. This part is also segmented and vectorized using the Tokenizer method. According to the task, the mean square error is selected as the loss function and the loss is calculated. The gradient of the loss to the model parameters is calculated by the back propagation algorithm, and AdmaW is used to update the model parameters. This process will iterate multiple training samples until the specified training rounds are reached or the convergence condition is reached.

[0122] Embodiment 2:

[0123] The embodiment of the present invention provides an input coding system of a pre-trained network based on a code hierarchy, comprising:

[0124] The word segmentation module is used to segment the original code sequence and construct the corresponding token vocabulary;

[0125] A conversion module, used to convert the code sequence after word segmentation using the token vocabulary to obtain the code sequence embedding;

[0126] An acquisition module, used for acquiring a level value sequence embedding according to the indentation level of the original code sequence;

[0127] A first encoding module is used to obtain a hierarchical encoding result according to the hierarchical value sequence embedding; and to concatenate the hierarchical value sequence embedding and the code sequence embedding to obtain a vector representation result;

[0128] A second encoding module, used to obtain the position encoding result of the original code sequence;

[0129] A fusion module is used to fuse the vector representation result, the hierarchical encoding result and the position encoding result to obtain an input vector of the pre-trained network.

[0130] Embodiment 3:

[0131] An embodiment of the present invention provides a storage medium storing a computer program for input encoding of a pre-trained network based on a code hierarchy, wherein the computer program enables a computer to execute the input encoding method as described above.

[0132] Embodiment 4:

[0133] An embodiment of the present invention provides an electronic device, including:

[0134] One or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the input encoding method as described above.

[0135] It can be understood that the input encoding system, storage medium and electronic device of the pre-trained network based on the code hierarchy structure provided in the embodiment of the present invention correspond to the input encoding method of the pre-trained network based on the code hierarchy structure provided in the embodiment of the present invention. The explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the input encoding method and will not be repeated here.

[0136] In summary, compared with the prior art, the present invention has the following beneficial effects:

[0137] 1. The embodiment of the present invention uses a code hierarchy to improve the encoding method of the pre-trained network input. Compared with the prior art, this method can better capture the structure and semantic information of the code and improve the accuracy and effect of the encoding.

[0138] 2. The embodiment of the present invention introduces hierarchical information into the encoding process of the code sequence and combines it with the embedding of the token sequence, so that the vectorization of the code sequence retains more structural information, while many current language models do not pay attention to the structural information of the code.

[0139] 3. The encoding method provided by the embodiment of the present invention can introduce more semantic information into the pre-trained network input, and improve the model's ability to understand and express the code. Compared with the manually designed rules in the prior art, this method can enhance the model's ability to generalize the code and improve the model's adaptability and performance in different tasks and fields.

[0140] 4. The encoding method of the present invention can make full use of the characteristics of the code hierarchy structure, combine the hierarchy information with the vectorized code sequence, and convert it into a more compact representation. Compared with the existing method of obtaining global hierarchy information, this method can reduce the storage space and computing resources required for encoding, and improve the efficiency and speed of encoding.

[0141] 5. The encoding method of the present invention is applicable to multiple fields, such as code summarization, code translation, etc. In these fields, the encoding method of the present invention can improve the performance and effect of the model and better meet actual needs.

[0142] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0143] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for encoding input of a pre-trained network based on a code hierarchy, characterized in that: include: Segment the original code sequence and construct the corresponding token vocabulary; Using the token vocabulary to transform the code sequence after word segmentation to obtain the code sequence embedding; According to the indentation level of the original code sequence, obtaining a level value sequence embedding; Embedding the hierarchical value sequence to obtain a hierarchical coding result; and concatenating the hierarchical value sequence embedding of each row with the code sequence embedding to which the row belongs, to obtain a vector representation result; Obtaining a position encoding result of the original code sequence; Fusing the vector representation result, the hierarchical encoding result and the position encoding result to obtain an input vector of the pre-trained network; The step of acquiring a level value sequence embedding according to the indentation level of the original code sequence comprises: The original code sequence is formatted normally and traversed line by line, and the indentation of each line is used as the level value to obtain the code sequence S token The corresponding level value sequence S level ; For the level value sequence S level deal with: E level =Embedding′(S level ,d) Among them, E level represents hierarchical value sequence embedding, and Embedding′ represents hierarchical embedding operation; The hierarchical value sequence is embedded and normalized to obtain the hierarchical encoding result, which is expressed as: E′ level =Sigmoid(E level ) Among them, E′ level Represents the hierarchical encoding result, Sigmoid represents the nonlinear activation function; The hierarchical value sequence embedding of each row is concatenated with the code sequence embedding to which the row belongs to obtain a vector representation result; it is expressed as: E′ token =Linear(concat(E level ,E token ,0),k*d,d) Among them, E′ token Indicates vector representation result; Linear indicates linear transformation operation; concat indicates concatenation operation; E token Represents a code sequence embedding.

2. The input coding method according to claim 1, characterized in that: The method of segmenting the original code sequence and constructing a corresponding vocabulary includes: Selecting a syntax analyzer corresponding to the programming language used by the original code sequence according to the language type contained in the data sample; Load the local parser and the corresponding language; The original code sequence is segmented by the loaded syntax analyzer, and a corresponding token vocabulary is constructed; each token group contains a number of sub-tokens.

3. The input coding method according to claim 2, characterized in that: The method of converting the segmented code sequence using the token vocabulary to obtain the code sequence embedding includes: The token word list is used to replace the code sequence after word segmentation, and the token is filled so that each token group contains k sub-tokens to obtain the code sequence S token ; For the code sequence S token Processing is performed to obtain the code sequence embedded in E token ; Among them, token i,j Represents the code sequence S token The jth subtoken in the i-th token, n token It represents the range of token values ​​contained in the token vocabulary, Embedding represents the code embedding operation, and d represents the embedding dimension.

4. The input coding method according to claim 1, characterized in that: The position encoding result PE of the original code sequence is obtained by adopting the cosine position encoding method.

5. The input coding method according to claim 4, characterized in that: The fusion of the vector representation result, the hierarchical encoding result and the position encoding result to obtain the input vector of the pre-trained network is expressed as: input=W h ·HAVE BEEN' level +W p ·PE+E' token Among them, input represents the input vector, W h , W p Represent the corresponding weights respectively.

6. An input encoding system for a pre-trained network based on a code hierarchy, characterized in that: Used to perform the input encoding method as claimed in claim 1, comprising: The word segmentation module is used to segment the original code sequence and construct the corresponding token vocabulary; A conversion module, used to convert the code sequence after word segmentation using the token vocabulary to obtain the code sequence embedding; An acquisition module, used for acquiring a level value sequence embedding according to the indentation level of the original code sequence; A first encoding module is used to obtain a hierarchical encoding result according to the hierarchical value sequence embedding; and to concatenate the hierarchical value sequence embedding of each row with the code sequence embedding to which the row belongs to obtain a vector representation result; A second encoding module, used to obtain a position encoding result of the original code sequence; A fusion module is used to fuse the vector representation result, the hierarchical encoding result and the position encoding result to obtain an input vector of the pre-trained network.

7. A storage medium, characterized in that: It stores a computer program for input encoding of a pre-trained network based on a code hierarchy, wherein the computer program enables a computer to execute the input encoding method as claimed in any one of claims 1 to 5.

8. An electronic device, characterized in that: include: one or more processors; Memory; and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the programs including instructions for executing the input encoding method according to any one of claims 1 to 5.