Training methods, code completion methods, and devices for code completion models

Abstract

This application provides a training method, a code completion method, and an apparatus thereof for a code completion model, relating to the field of artificial intelligence. The training method for the code completion model includes: acquiring first training data, the first training data including program code, the program code including a sequence of lexical terms represented using a two-line parallel sequence, wherein the two-line parallel sequence includes a type-line and a token-line, the type-line indicating the structured syntactic information of the program code, and the token-line indicating the semantic information of the program code; inputting the first training data into a neural network model and training the neural network model based on a custom Transformer multilayer neural network to obtain a target code completion model. In this scheme, the target code completion model can generate complete lines of code end-to-end, with accurate prediction and fast inference, helping to improve user development efficiency and experience.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a training method for a code completion model, a code completion method, and an apparatus thereof. Background Technology

[0002] Code completion technology has always been a research hotspot in the field of software engineering. This technology greatly reduces the workload of programmers and improves development efficiency, quality and experience. Currently, all mainstream integrated development environments (IDEs) include code auto-completion as a core function.

[0003] Traditional IDE code completion systems only consider the type compatibility and referenceability of application programming interfaces (APIs). When faced with complex software frameworks, these systems often have low accuracy in recommending APIs. The main reason is that these recommendation methods simply filter all APIs based on static analysis results, recommending a large number of methods or fields, and finally providing the recommendations by alphabetical sorting or frequency statistics.

[0004] Currently, there is no mature solution for intelligent code line completion technology in the industry. This is mainly because programming languages ​​have many characteristics that are different from natural languages, such as the out-of-vocabulary (OOV) problem, diversity problem, and long-range dependency problem. Traditional natural language processing technology faces many difficulties. At the same time, for the sake of code privacy protection, deep models need to be deployed on local processors, which places more stringent requirements on the runtime efficiency of the algorithm. Summary of the Invention

[0005] This application provides a training method for a code completion model, a code completion method, and an apparatus thereof, which helps to achieve accurate and fast code completion.

[0006] In a first aspect, a training method for a code completion model is provided, comprising: acquiring first training data, the first training data including program code, the program code including a sequence of lexical terms represented by a double-line parallel sequence, wherein the double-line parallel sequence includes a type line and a token line, the type line being used to indicate the structured syntactic information of the program code, and the token line being used to indicate the semantic information of the program code; inputting the first training data into a neural network model and training the neural network model based on a multi-layer neural network of a custom Transformer to obtain a target code completion model.

[0007] According to the technical solution of this application, the program code is represented by a two-line parallel sequence, which has better sample distribution quality and can better support the subsequent inference process to accelerate model training.

[0008] Furthermore, the target code completion model obtained according to the technical solution of this application can generate complete lines of code end-to-end, with accurate prediction and fast inference, which helps to improve user development efficiency and experience.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the above-mentioned input of the first training data into the neural network model includes: constructing a vocabulary, the vocabulary including a type vocabulary and a token vocabulary, the type vocabulary being constructed from the complete set of type vocabulary, and the key values ​​corresponding to each index in the token vocabulary being not fixed; mapping the word sequence in the first training data to an integer sequence according to the vocabulary, and inputting the integer sequence into the neural network model.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the lexical sequence in the first training data includes OOV words outside the vocabulary list, and the above method further includes: filling in the OOV words sequentially from the head of the Token vocabulary list in order of distance from the current cursor position; and / or, filling in the OOV words sequentially from the tail of the Token vocabulary list.

[0011] According to the technical solution of this application, the OOV problem is solved simply and efficiently by introducing a dynamic position vocabulary, which helps to quickly and accurately train the target code completion model.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the above-mentioned Transformer-based multilayer neural network training neural network model includes: embedding integer sequences representing Type and Token into high-dimensional real vector sequences, and then concatenating them into an 88-dimensional input vector sequence after encoding; using a first transform network to perform feature encoding on the input vector sequence to obtain a feature map; using a second transform network to perform syntactic decoding on the feature map, and using a third transform network to perform semantic decoding on the feature map to train the neural network; wherein the depth of the first transform network is greater than the depth of the second transform network and the depth of the third transform network, and the depth of the third transform network is greater than the depth of the second transform network.

[0013] According to the technical solution of this application, training a neural network model based on a multi-layer neural network of Transformer has better long-range dependency learning ability, and its easy parallel computing feature can also improve the speed of training target code to complete the model.

[0014] Optionally, when training the target model, multi-task joint learning can be used for training, which can speed up the model convergence and improve the model's generalization ability.

[0015] Secondly, a code completion method is provided, comprising: obtaining program code input by a user; inputting the program code into a target code completion model to obtain a code completion result, wherein the target code completion model is obtained by training a neural network model using a first training data and based on a multi-layer neural network structure of a modified Transformer; the first training data includes program code, wherein the program code includes a sequence of lexical terms represented by a two-line parallel sequence, wherein the two-line parallel sequence includes a type line and a token line, the type line is used to indicate the structured syntactic information of the program code, and the token line is used to indicate the semantic information of the program code.

[0016] According to the technical solution of this application, the target code completion model can generate complete lines of code end-to-end, with accurate prediction and fast reasoning, which helps to improve user development efficiency and experience.

[0017] In conjunction with the second aspect, in some implementations of the second aspect, the above-mentioned input of program code into the target code completion model to obtain code completion results includes: converting program code into a sequence of lexical terms represented using two-line parallel sequences; mapping the lexical sequence to an integer sequence based on a vocabulary, the vocabulary including a Type vocabulary and a Token vocabulary, the Type vocabulary being constructed from the complete set of Type vocabulary, and the key values ​​corresponding to each index in the Token vocabulary being variable; and obtaining the code completion result based on the integer sequence.

[0018] According to the technical solution of this application, the program code is represented by a two-line parallel sequence, which has better sample distribution quality and can be better used for reasoning to obtain code completion results.

[0019] Furthermore, according to the technical solution of this application, by introducing a dynamic position vocabulary, the OOV problem is solved simply and efficiently, which helps to obtain code completion results quickly and accurately.

[0020] In conjunction with the second aspect, in some implementations of the second aspect, obtaining the code completion result based on the integer sequence includes: performing single-step reasoning on the integer sequence representing Type to obtain the single-step reasoning result representing Type; performing single-step reasoning on the integer sequence representing Token to obtain the single-step reasoning result representing Token; and obtaining the code completion result based on the single-step reasoning result representing Type and the single-step reasoning result representing Token.

[0021] Optionally, the above method further includes: performing static analysis on the integer sequence to obtain static analysis results, and obtaining code completion results based on the single-step reasoning results representing Type and the single-step reasoning results representing Token, including: using the static analysis results to perform conditional pruning on the single-step reasoning results representing Type and the single-step reasoning results representing Token to obtain code completion results.

[0022] According to the technical solution of this application, the static analysis results are used to perform condition pruning on the results of single-step reasoning, which helps to ensure the grammatical correctness of the reasoning results. Furthermore, it can also improve the reasoning speed of the model.

[0023] In conjunction with the second aspect, in some implementations of the second aspect, the single-step reasoning result representing Type and the single-step reasoning result representing Token include multiple branch groups, the similarity of the output results of multiple branch groups is lower than a preset value, and the code completion result includes the output results of multiple groups.

[0024] According to the technical solution of this application, the output of results with similarity lower than the preset value is adopted in the form of multiple branch groups, which helps to improve the diversity of code completion results.

[0025] Thirdly, a training apparatus for a code completion model is provided, characterized in that it includes: an acquisition unit for acquiring first training data, the first training data including program code, the program code including a sequence of lexical terms represented by a two-line parallel sequence, wherein the two-line parallel sequence includes a type-line and a token-line, the type-line being used to indicate the structured grammatical information of the program code, and the token-line being used to indicate the semantic information of the program code; and a training unit for inputting the first training data into a neural network model and training the neural network model based on a multi-layer neural network of a custom Transformer to obtain a target code completion model.

[0026] In conjunction with the third aspect, in some implementations of the third aspect, the training unit is specifically used to: construct a vocabulary, which includes a Type vocabulary and a Token vocabulary. The Type vocabulary is constructed from the complete set of Type vocabulary, and the key values ​​corresponding to each index in the Token vocabulary are not fixed; map the word sequence in the first training data to an integer sequence according to the vocabulary, and input the integer sequence into the neural network model.

[0027] In conjunction with the third aspect, in some implementations of the third aspect, the training unit is also used to: fill in OOV words sequentially from the head of the Token vocabulary in order of distance from the current cursor position; and / or, fill in OOV words sequentially from the tail of the Token vocabulary.

[0028] In conjunction with the third aspect, in some implementations of the third aspect, the training unit is specifically used to: embed the integer sequences representing Type and Token into a high-dimensional real vector sequence, and then concatenate them into an 88-dimensional input vector sequence after encoding; use a first transformation network to perform feature encoding on the input vector sequence to obtain a feature map; use a second transformation network to perform syntactic decoding on the feature map, and use a third transformation network to perform semantic decoding on the feature map to train a neural network model; wherein, the depth of the first transformation network is greater than the depth of the second transformation network and the depth of the third transformation network, and the depth of the third transformation network is greater than the depth of the second transformation network.

[0029] Fourthly, a code completion apparatus is provided, comprising: an acquisition unit for acquiring program code input by a user; and a processing unit for inputting the program code into a target code completion model to obtain a code completion result. The target code completion model is obtained by training a neural network model using first training data and based on a multi-layer neural network structure of a modified Transformer. The first training data includes program code, which includes a sequence of lexical terms represented by a two-line parallel sequence. The two-line parallel sequence includes a type line and a token line. The type line is used to indicate the structured syntactic information of the program code, and the token line is used to indicate the semantic information of the program code.

[0030] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is specifically used to: convert program code into a sequence of lexical terms represented by a two-line parallel sequence; map the lexical sequence into an integer sequence based on a vocabulary, which includes a Type vocabulary and a Token vocabulary. The Type vocabulary is constructed from the complete set of Type vocabulary, and the key values ​​corresponding to each index in the Token vocabulary are not fixed; and obtain the code completion result based on the integer sequence.

[0031] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is specifically used to: perform single-step reasoning on the integer sequence representing Type to obtain the single-step reasoning result representing Type; perform single-step reasoning on the integer sequence representing Token to obtain the single-step reasoning result representing Token; and obtain the code completion result based on the single-step reasoning result representing Type and the single-step reasoning result representing Token.

[0032] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the processing unit is also used to: perform static analysis on the integer sequence to obtain static analysis results, and obtain code completion results based on the single-step reasoning results representing Type and the single-step reasoning results representing Token, including: using the static analysis results to perform conditional pruning on the single-step reasoning results representing Type and the single-step reasoning results representing Token to obtain code completion results.

[0033] In conjunction with the fourth aspect, in some implementations of the fourth aspect, the single-step reasoning result representing Type and the single-step reasoning result representing Token include multiple branch groups, the similarity of the output results of multiple branch groups is lower than a preset value, and the code completion result includes the output results of multiple groups.

[0034] Fifthly, a training apparatus for a code completion model is provided, the training apparatus including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the training method described in the first aspect.

[0035] In a sixth aspect, a code completion device is provided, the device including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method described in the second aspect.

[0036] A seventh aspect provides a computer-readable storage medium, characterized in that the computer-readable medium stores program code for execution by a device, the program code including instructions for performing the method described in the first or second aspect.

[0037] Eighthly, a computer program product is provided, characterized in that, when the computer program is executed on a computer, it causes the computer to perform the method described in the first aspect or the second aspect. Attached Figure Description

[0038] Figure 1 This is a schematic diagram of an artificial intelligence main framework according to this application.

[0039] Figure 2 This is a schematic diagram of an existing code completion method based on rule templates.

[0040] Figure 3 This is a schematic diagram of a system architecture according to an embodiment of this application.

[0041] Figure 4 This is a schematic diagram of the structure of a convolutional neural network according to this application.

[0042] Figure 5 This is a schematic diagram of the hardware structure of a chip according to this application.

[0043] Figure 6 This is an example of a flowchart illustrating the training method for the code completion model in this application.

[0044] Figure 7 This is a schematic diagram of a network structure to which this application applies.

[0045] Figure 8 This is an example flowchart illustrating the code completion method in this application.

[0046] Figure 9 This is a schematic block diagram of the training device for the code completion model of this application.

[0047] Figure 10 This is a schematic diagram of the hardware structure of the training device for the code completion model of this application.

[0048] Figure 11 This is a schematic block diagram of the code completion device of this application.

[0049] Figure 12 This is a schematic diagram of the hardware structure of the code completion device in this application. Detailed Implementation

[0050] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0051] Figure 1 This is a schematic diagram of an artificial intelligence main framework according to an embodiment of this application. The main framework describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field needs.

[0052] The above-mentioned artificial intelligence framework will be elaborated in detail from two dimensions: the "intelligent information chain" (horizontal axis) and the "information technology (IT) value chain" (vertical axis).

[0053] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0054] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0055] (1) Infrastructure:

[0056] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0057] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0058] The intelligent chip here can be a hardware acceleration chip such as a central processing unit (CPU), a neural network processing unit (NPU), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0059] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0060] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0061] (2) Data:

[0062] The data at the layer above the infrastructure is used to represent data sources in the field of artificial intelligence. This data involves at least one type of information, such as graphics, images, voice, and text. This data varies across different application areas and can have different forms of representation. For example, in the context of the Internet of Things (IoT), the content of this data is related to specific IoT connected terminals and may include sensor data such as force, displacement, liquid level, temperature, or humidity.

[0063] In the embodiments of this application, the data is, for example, code data, which may be a code database obtained by crawling from a website.

[0064] (3) Data processing:

[0065] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0066] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0067] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0068] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0069] (4) General abilities:

[0070] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0071] (5) Smart products and industry applications:

[0072] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, safe city, or intelligent terminals, etc.

[0073] The embodiments of this application can be applied to the field of intelligent code completion, such as program development and program modification.

[0074] Currently, there are various code completion methods available. The following is a brief introduction to the existing code completion methods.

[0075] (1) The most widely used intelligent code completion technology is based on a machine learning classification model. Representative products using this technology include the official VS Code plugin IntelliCode and the third-party commercial software Kite. This method first extracts specific features through feature engineering based on expert knowledge, then builds a machine learning classification model based on the selected features, and uses the prediction results of the classification model to complete the subsequent code. A typical code completion scenario for this technology is, for example, when typing "." after "if path", the technology can pop up a completion list providing options such as "StartsWith", "Length", "Replace", and "EndsWith" for the user to choose from. However, this technology is limited by the weak expressive power of the machine learning model, which cannot learn complex semantics and norms. Furthermore, the classification model does not have the ability to learn the positional relationship of sequences, and the prediction effect is very poor for more than one token. In addition, this method requires triggering multiple predictions and multiple user selections to complete the entire line, which is inefficient and results in a poor user experience.

[0076] (2) Code completion method based on rule templates. For example... Figure 2 As shown, this method first determines the intent of the current statement based on rule-driven or data-driven approaches. Then, it generates a statement template with blank slots according to the rules, and finally uses a deep learning model to fill the blank slots, completing the entire line completion. However, formulating rule templates is very time-consuming and the templates lack universality. When the language is switched or updated, the workload for porting is large. Furthermore, since the completion is based on rule templates, the completion results are relatively mechanical and fixed, with weak ability to capture contextual semantics. When the context changes, the completion results of this technique may not be updated in time to meet the current context. In addition, it is obvious that this method is limited by rule templates, and the completion effect of statements outside the templates is extremely poor. It also cannot effectively learn code style and programming patterns.

[0077] (3) Code completion method based on deep learning sequence generation model. This method uses a seq2seq model to directly map the input code fragment into a continuous code sequence, enabling arbitrary line completion at any position. Chinese Invention Patent Application No. 201810231329.7 proposes a code completion method based on Long Short-Term Memory (LSTM) network. Addressing the shortcomings of existing code completion techniques, such as low accuracy in line completion and lack of effective integration with syntax tree information, this method uses an abstract syntax tree (AST) to parse the source code, then trains a language model using LSTM, and finally uses a character-level LSTM during inference and prediction to achieve code completion. The feature extractor used in this method is the LSTM temporal network, whose biggest drawback is its inability to efficiently perform parallel computation and parameter reuse, thus limiting its ability to handle large-scale vocabularies. Therefore, this method cannot sample token-level vocabularies and instead uses a character-level vocabulary with identifiers. However, due to the character-level discontinuity inherent in code languages ​​(i.e., a change in a single character within a line of code can lead to a semantic "jump"), this method cannot accurately learn code sequences end-to-end. It relies heavily on rule-based post-processing to ensure the reasonableness of the generated results, increasing the method's coupling and reducing actual prediction accuracy. Furthermore, LSTM networks cannot handle large-scale dependencies in sequences, which are particularly common in code languages ​​(e.g., the definition of a variable or the previous branch of a logical statement can be traced back dozens of lines). Model inference is also time-consuming, making local deployment impossible. Most importantly, this method does not effectively incorporate static analysis for syntax filtering, resulting in numerous syntax errors in the completion results.

[0078] For the reasons mentioned above, this application proposes a training method for a code completion model, a method for code completion, and an apparatus for code completion, which helps to achieve accurate and rapid code completion.

[0079] Figure 3 This is a schematic diagram of a system architecture according to an embodiment of this application, which can be used to train neural network models, such as code completion models. Figure 3 As shown, the data acquisition device 160 is used to collect training data. For the method of this embodiment, the training data may include training sequences and corresponding completion results, wherein the completion results of the training sequences may be manually pre-annotated results. For the training of the code completion model in this embodiment, the training sequence may be an integer sequence that maps morpheme sequences to a pre-constructed dynamic vocabulary.

[0080] After collecting the training data, the data acquisition device 160 stores the training data in the database 130, and the training device 120 trains the target model / rule 101 based on the training data maintained in the database 130. "A / B" describes the relationship between related objects, indicating that there can be three kinds of relationships. For example, A / B can represent three cases: A exists alone, A and B exist simultaneously, and B exists alone.

[0081] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. In one case, the training device 120 processes the category feature vector of the input sequence, compares the output category with the label category, and continues until the accuracy of the category output by the training device 120 is greater than or equal to a certain threshold, thereby completing the training of the target model / rule 101. In this case, the code completion model of this application embodiment can be trained.

[0082] The target model / rule 101 described above can be used to implement the method of the embodiments of this application. Specifically, the target model / rule 101 in the embodiments of this application can be a neural network. It should be noted that in practical applications, the training data maintained in the database 130 may not all come from the data acquisition device 160; it may also be received from other devices. Furthermore, it should be noted that the training device 120 may not necessarily train the target model / rule 101 entirely based on the training data maintained in the database 130; it may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application.

[0083] The target model / rule 101 trained using training device 120 can be applied to different systems or devices, such as... Figure 3The execution device 110 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, augmented reality (AR) / virtual reality (VR) device, vehicle terminal, etc., or it can be a server or cloud service. Figure 3 In this embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140. The input data may include the original sequence input by the client device in this application embodiment.

[0084] Preprocessing module 113 and preprocessing module 114 are used to preprocess the input data (such as the original sequence) received by I / O interface 112. In this embodiment, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of them may be used), and the calculation module 111 may be used directly to process the input data.

[0085] During the preprocessing of input data by the execution device 110, or during the calculation module 111 of the execution device 110 performing calculations and other related processes, the execution device 110 can call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0086] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0087] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different objectives or tasks. The corresponding target models / rules 101 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0088] exist Figure 3In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If prior authorization from the user is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can be presented in the form of display or actions. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0089] It is worth noting that, Figure 3 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 3 In this case, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0090] like Figure 3 As shown, the target model / rule 101 is trained by the training device 120. The target model / rule 101 can be a neural network obtained by the method of the embodiment of this application. Specifically, the neural network of the embodiment of this application can be a convolutional neural network (CNN) that can be used for code completion, or a deep convolutional neural network (DCNN), etc.

[0091] Since CNN is a very common type of neural network, and is the neural network that this application focuses on in its embodiments, the following will be combined with... Figure 4 This section focuses on a detailed explanation of the structure of CNNs. As mentioned in the basic concept introduction above, a Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, where each neuron can respond to the input sequence.

[0092] In one implementation, the underlying neural network in the code completion method of this application embodiment specifically adopts a neural network structure as follows: Figure 4 As shown.

[0093] Figure 4 This is a schematic diagram of the structure of a convolutional neural network. Figure 4 In this model, the Convolutional Neural Network (CNN) 200 may include an input layer 210, a layer 220 (layer 220 may include convolutional layers and pooling layers, or layer 220 may include convolutional layers but not pooling layers), and a fully connected layer 230. The input layer 210 can obtain user-input code or project file code, and then process the obtained code to be completed by layer 220 and the subsequent fully connected layer 230 to obtain the code completion result. The following section... Figure 4 This section provides a detailed introduction to the internal layer structure of CNN 200.

[0094] Floor 220:

[0095] Convolutional layers:

[0096] by Figure 4 For example, Figure 4 Layer 220 shown may include layers as in Examples 221-226. For instance, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation. The number and position of convolutional and pooling layers shown here are merely examples; there may be more or fewer convolutional and pooling layers, or no pooling layers at all.

[0097] The following section will use convolutional layer 221 as an example to introduce the internal working principle of a convolutional layer.

[0098] Convolutional layer 221 can include many convolution operators, also known as kernels. In code processing, a convolution operator acts as a filter to extract specific information from the input program code. A convolution operator can essentially be a weight matrix, which is usually predefined. During the convolution operation on the program code, the weight matrix can usually parse a piece of program code into a syntax tree. Then, the syntax tree is processed by filtering low-semantic nodes, subdividing rich-semantic nodes, compressing redundant nodes, extracting structured information, and separating system libraries, thereby improving data quality. Finally, a depth-first traversal can be used to obtain the node sequence.

[0099] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input code, thereby enabling the convolutional neural network 200 to make correct predictions.

[0100] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0101] Pooling layer:

[0102] Since it is often necessary to reduce the number of training parameters, pooling layers can be periodically introduced after convolutional layers, such as... Figure 4 In the example of 220, each layer 221-226 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers.

[0103] Fully connected layer 230:

[0104] After processing by layer 220, the convolutional neural network 200 is still insufficient to output the required output information. To generate the final output information (the required class information or other relevant information), the convolutional neural network 200 further utilizes fully connected layers 230 to generate one or a set of outputs representing the required number of classes. Therefore, the fully connected layers 230 may include multiple hidden layers (such as...). Figure 4 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as OOV problems, syntax checking, diversity, etc.

[0105] After the multiple hidden layers in the fully connected layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has propagated forward (e.g., ... Figure 4 Propagation from 210 to 240 degrees is considered forward propagation, while backward propagation (e.g.) is completed. Figure 4 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0106] It should be noted that, Figure 4 The convolutional neural network shown is only one possible example of a convolutional neural network that forms the basis of the code completion method in this application embodiment. In specific applications, the convolutional neural network used as the basis of the code completion method in this application embodiment can also exist in the form of other network models.

[0107] Figure 5 This is a schematic diagram of the hardware structure of a chip according to an embodiment of this application. The chip includes a neural network processor (NPU300 shown). This chip can be configured as follows: Figure 3 The execution device 110 shown is used to perform the calculations of the calculation module 111. This chip can also be located in, for example... Figure 3 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101. For example... Figure 4 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 5 This is achieved in the chip shown.

[0108] The NPU300 is mounted as a coprocessor on the main central processing unit (CPU) (host CPU), and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 303, and the controller 304 controls the arithmetic circuit 303 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0109] In some implementations, the arithmetic circuit 303 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 303 is a two-dimensional pulsating array. The arithmetic circuit 303 can also be a one-dimensional pulsating array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 303 is a general-purpose matrix processor.

[0110] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 302 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 301 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 308.

[0111] The vector computation unit 307 can further process the output of the arithmetic circuit, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 307 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0112] In some implementations, the vector computation unit 307 can store the processed output vector into a unified buffer 305. For example, the vector computation unit 307 can apply a nonlinear function to the output of the arithmetic circuit 303, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 307 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as activation input to the arithmetic circuit 303, for example, for use in subsequent layers of a neural network.

[0113] The unified memory 305 is used to store input data and output data.

[0114] The weight data is directly transferred from the external memory to the input memory 301 and / or the unified memory 305 through the direct memory access controller 305 (DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 305 is stored in the external memory.

[0115] The bus interface unit (BIU) 310 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 309 via a bus.

[0116] The instruction fetch buffer 509, connected to the controller 304, is used to store the instructions used by the controller 504.

[0117] The controller 304 is used to call the instructions cached in the instruction memory 309 to control the operation of the computing accelerator.

[0118] Entry point: The data here is explanatory data, such as the input program code, etc., as can be described in the actual description.

[0119] Optionally, the unified memory 305, input memory 301, weighted memory 302, and instruction fetch memory 309 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0120] in, Figure 4 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 303 or the vector calculation unit 307.

[0121] The above-mentioned Figure 3 The execution device 110 in the embodiment is capable of executing each step of the code completion method or code completion model training method of the present application embodiments. Figure 4 The CNN model shown and Figure 5 The chip shown can also be used to execute various steps of the code completion method in the embodiments of this application.

[0122] Figure 6 This is an example of a schematic flowchart illustrating the training method of the code completion model according to an embodiment of this application. The following is a description of... Figure 6 The process will be described step by step.

[0123] S410, acquire the first training data.

[0124] The first training data includes program code, which can be a sequence of lexical terms represented by a double-line parallel sequence after processing. The double-line parallel sequence includes a type-line and a token-line. The type-line is used to indicate the structured syntax information of the program code, and the token-line is used to indicate the semantic information of the program code.

[0125] Specifically, a large-scale code corpus can be obtained. First, it is parsed into a Program Structure Interface (PSI) syntax tree. Then, a series of preprocessing operations are performed on the PSI tree to improve data quality, including filtering low-semantic nodes, subdividing rich-semantic nodes, compressing redundant nodes, extracting structured information, and separating system libraries. Finally, a depth-first traversal is used to obtain a node sequence, where each node contains two fields: Type and Token (for nodes without a Token field, the Token is manually marked as "Empty"). This results in a two-line parallel sequence structure containing Type-line and Token-line. The Type-line represents the structured syntactic information of the code, and the Token-line represents the semantic information of the code.

[0126] According to the technical solution of this application, the program code is represented by a two-line parallel sequence, which has better sample distribution quality and can better support the subsequent inference process to accelerate model training.

[0127] S420: Input the first training data into the neural network model and train the neural network model based on a custom Transformer multilayer neural network to obtain the target code completion model.

[0128] The above-mentioned inputting the first training data into the neural network model may further include: constructing a vocabulary, which includes a Type vocabulary and a Token vocabulary. The Type vocabulary is constructed from the complete set of Type vocabulary, and the key values ​​corresponding to the indexes in the Token vocabulary are not fixed; mapping the word sequence in the first training data to an integer sequence according to the vocabulary, and inputting the integer sequence into the neural network model.

[0129] Optionally, the morpheme sequence in the first training data includes OOV words outside the vocabulary list, and the above method further includes: filling in the OOV words sequentially from the head of the Token vocabulary list in order of distance from the current cursor position; and / or filling in the OOV words sequentially from the tail of the Token vocabulary list.

[0130] Specifically, since the Type vocabulary does not have an OOV (Out of Context) issue, it can be directly constructed from the complete Type vocabulary set. In this embodiment, the Type vocabulary has a dimension of 185. For the Token vocabulary, it can be divided into a static vocabulary and a dynamic vocabulary. In this embodiment, the index range of the static vocabulary is [0, 30000], and the index range of the dynamic vocabulary is [30001, 30500]. The key values ​​of each index in the static vocabulary are fixed, mainly composed of built-in language vocabulary (such as keywords, operators, etc.), built-in library vocabulary (such as class names, method names, and variable names built into the Java JDK), high-frequency vocabulary from the training corpus, and auxiliary vocabulary (such as "pad", "Empty", "UNK", etc.). Unlike the static vocabulary, the key values ​​corresponding to each index in the dynamic vocabulary are not fixed. Instead, they are dynamically filled into the vocabulary based on certain rules, using OOV vocabulary appearing in the runtime context and dependency files. The specific rules are as follows:

[0131] a) For OOV words introduced by the context, fill them in sequentially from the head of the dynamic word list (position 30001 in this example) in order of distance from the current cursor position. For example, "stringBuilder" will be filled in position 30001, "string" will be filled in position 30002, and so on.

[0132] b) For OOV words introduced by the dependency file, fill them in in any order from the end of the dynamic vocabulary (i.e., position 30500 in this embodiment).

[0133] c) Stop filling all OOV vocabulary or when the dynamic vocabulary is completely filled.

[0134] By setting up a dynamic vocabulary, all OOV (Out-of-Vocabulary) words can be represented accordingly. Simultaneously, for words at the head of the dynamic vocabulary, the model can learn their semantics through their relative positions. For example, index 30001 always points to the first OOV word before the cursor (in this example, the position of "stringBuilder"), and 30002 always points to the second OOV word before the cursor (in this example, the position of "string"). Thus, the model can learn feature representations for these positions rather than individual words (acting like "pointers") based on the words appearing before and after these positions and their relative positions. Therefore, regardless of how the user customizes and names these variables, the model can always provide the accurate prediction shown in the figure.

[0135] It should be noted that since this method only adds a dynamic vocabulary construction operation in the preprocessing stage, the additional computational cost in the runtime inference stage comes only from expanding the vocabulary from 30,000 dimensions to 30,500 dimensions, which is almost negligible in terms of the increase in computational burden.

[0136] According to the technical solution of this application, the OOV problem is solved simply and efficiently by introducing a dynamic position vocabulary, which helps to quickly and accurately train the target code completion model.

[0137] Optionally, the above-mentioned Transformer-based multilayer neural network training neural network model includes: embedding integer sequences representing Type and Token into high-dimensional real vector sequences, and then concatenating them into an 88-dimensional input vector sequence after encoding; using a first transform network to perform feature encoding on the input vector sequence to obtain a feature map; using a second transform network to perform syntactic decoding on the feature map, and using a third transform network to perform semantic decoding on the feature map to train the neural network; wherein the depth of the first transform network is greater than the depth of the second transform network and the depth of the third transform network, and the depth of the third transform network is greater than the depth of the second transform network.

[0138] Specifically, the network structure used in the embodiments of this application is as follows: Figure 7 As shown, it mainly includes the following parts:

[0139] (1) Embedding layer

[0140] This part can embed the integer sequences representing Type and Token into high-dimensional real vector sequences using two trainable Embedding matrices (185x16 and 30500x72 in this embodiment, respectively), and then concatenate them into an 88-dimensional input vector sequence after encoding by relative position.

[0141] (2) Encoder layer

[0142] This section uses a very deep first transform network, Transformer1 (in this embodiment, a serial Transformer structure with a depth of 8; where 8 blocks and 8 heads do not have a completely corresponding Chinese meaning, but can be understood as 8 blocks and 8 heads) to encode the features of the input vector sequence. Deeper networks offer better learning capabilities but also incur a greater computational burden. Using a network with a depth of 8 as the encoding layer satisfies the need for feature extraction from a wide range of contextual information while also ensuring inference efficiency.

[0143] (3) Decoder layer

[0144] Since the representation complexity of grammatical information is much lower than that of semantic information (specifically reflected in the vocabulary dimension of Type being 185, while the vocabulary dimension of Token is 30500), this part can first use a very shallow second transformation network Transformer2 (a Transformer structure with a depth of 1 in this embodiment) to perform grammatical decoding on the feature map, and then use a deeper third transformation network Transformer3 (a serial Transformer structure with a depth of 4 in this embodiment) to perform semantic decoding.

[0145] (4) Specific task layer

[0146] This part can perform multiple related subtasks simultaneously and generate various task-related outputs, as shown in the figure. Here, "N_preds" is the prediction for Type, "T_preds" is the prediction for Token, "Loss-T-non-blacklist" represents the Loss value of the token morpheme, "Loss-T-identifier" represents the Loss value of the identifier token morpheme, and "Loss-N" represents the Loss value of the type morpheme.

[0147] According to the technical solution of this application, training a neural network model based on a multi-layer neural network of Transformer has better long-range dependency learning ability, and its easy parallel computing feature can also improve the speed of training target code to complete the model.

[0148] Meanwhile, when training the target model, multi-task joint learning can be used to accelerate the model's convergence speed and improve its generalization ability.

[0149] The target code completion model obtained by the technical solution of this application can generate complete lines of code end-to-end, with accurate prediction and fast reasoning, which helps to improve user development efficiency and experience.

[0150] Figure 8 This is an example of a schematic flowchart of the code completion method according to an embodiment of this application. The following is a description of... Figure 8 The process will be described step by step.

[0151] S610, obtains program code input by the user.

[0152] The code input by the user includes real-time input code and project files.

[0153] S620 inputs the program code into the target code completion model and obtains the code completion result.

[0154] The target code completion model is obtained by training a neural network model using the first training data and a multi-layer neural network structure based on the modified Transformer. The first training data includes program code, which includes a sequence of lexical terms represented by a two-line parallel sequence. The two-line parallel sequence includes a type line and a token line. The type line is used to indicate the structured syntax information of the program code, and the token line is used to indicate the semantic information of the program code.

[0155] Optionally, the above-mentioned input of program code into target code completion model to obtain code completion results includes: converting program code into a sequence of lexical terms represented by two-line parallel sequences; mapping the lexical sequence into an integer sequence based on a vocabulary, the vocabulary including a Type vocabulary and a Token vocabulary, the Type vocabulary being constructed from the complete set of Type vocabulary, and the key values ​​corresponding to each index in the Token vocabulary being non-fixed; and obtaining the code completion result based on the integer sequence.

[0156] Specifically, the program code is converted into a sequence of lexical terms, and then the lexical sequence is mapped into a sequence of integers to obtain the code completion result. Please refer to [link / reference needed] for details. Figure 6 The description in the text will not be repeated here.

[0157] According to the technical solution of this application, the program code is represented by a two-line parallel sequence, which has better sample distribution quality and can be better used for reasoning to obtain code completion results.

[0158] Furthermore, according to the technical solution of this application, by introducing a dynamic position vocabulary, the OOV problem is solved simply and efficiently, which helps to obtain code completion results quickly and accurately.

[0159] Optionally, obtaining the code completion result based on the integer sequence includes: performing single-step reasoning on the integer sequence representing Type to obtain a single-step reasoning result representing Type; performing single-step reasoning on the integer sequence representing Token to obtain a single-step reasoning result representing Token; and obtaining the code completion result based on the single-step reasoning result representing Type and the single-step reasoning result representing Token.

[0160] Specifically, during the inference process, the simultaneous execution of Type-line and Token-line inference is no longer adopted. Instead, a Type / Token separate inference mode is executed, and the specific process is as follows:

[0161] a) Perform Type-line inference and compute N_preds.

[0162] b) When the N_preds result belongs to the Identifier set (the set of Type types of nodes whose Token is the identifier, including TYPE_ID, VAR_ID, METHOD_ID, CLASS_ID, etc.), perform Token-line inference and calculate the corresponding T_preds.

[0163] c) When the result of N_preds does not belong to the Identifier set, it means that the current node is a keyword, operator or structure node. In this case, T_preds can be quickly obtained based on N_preds and the syntax rules.

[0164] According to the technical solution of this application, the use of a separate reasoning method significantly reduces the number of calculations, thereby significantly accelerating the speed of real-time reasoning.

[0165] Optionally, the above method further includes: performing static analysis on the integer sequence to obtain static analysis results, and obtaining code completion results based on the single-step reasoning results representing Type and the single-step reasoning results representing Token, including: using the static analysis results to perform conditional pruning on the single-step reasoning results representing Type and the single-step reasoning results representing Token to obtain code completion results.

[0166] Specifically, when predicting the next node of the sequence "if(file."), N_preds is first calculated to be "METHOD_ID", thus triggering Token-line inference. At this point, not all candidate words in the Token vocabulary are feasible; only some candidate words that satisfy the static analysis (e.g., method names that the variable "file" can reference, such as "exist" and "getPath") are feasible. Based on the index values ​​of these feasible words, we can prune the Embedding layer and the final output classifier (Softmax) layer in the original network, that is, the corresponding matrix only retains the dimension corresponding to the index values ​​of feasible words. This method ensures that the model only produces prediction results that satisfy the grammatical constraints of static analysis. On the other hand, since the matrix multiplication operation of the Softmax layer accounts for about 40% of the total computational cost of the model, and this method reduces the dimension of the Softmax layer from 30,500 to several hundred, this will greatly accelerate the inference speed of the model.

[0167] According to the technical solution of this application, the static analysis results are used to perform condition pruning on the results of single-step reasoning, which helps to ensure the grammatical correctness of the reasoning results. Furthermore, it can also improve the reasoning speed of the model.

[0168] Optionally, in this embodiment, the single-step reasoning result representing Type and the single-step reasoning result representing Token include multiple branch groups, and the similarity of the output results of the multiple branch groups is lower than a preset value. The code completion result includes the output results of multiple groups. In this way, using multiple branch groups to output results with a similarity lower than the preset value helps to improve the diversity of the code completion results.

[0169] According to the technical solution of this application, the target code completion model can generate complete lines of code end-to-end, with accurate prediction and fast reasoning, which helps to improve user development efficiency and experience.

[0170] Figure 9 This is a schematic block diagram of the training device for the code completion model in this application. Figure 9 The training device 700 for the code completion model shown includes an acquisition unit 701 and a training unit 702.

[0171] The acquisition unit 701 and the training unit 702 can be used to execute the training method of the code completion model of the present application embodiment. Specifically, the acquisition unit 701 can execute the above step S410, and the training unit 702 can execute the above step S420.

[0172] It should be understood that the training unit 702 in the above-described device 700 may be equivalent to the processor 802 in the device 800 described below.

[0173] Figure 10 This is a schematic diagram of the hardware structure of the training device for the code completion model in this application. Figure 10 The training device 800 for the code completion model shown (specifically, the device 800 can be a computer device) includes a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, processor 802, and communication interface 803 are interconnected via the bus 804.

[0174] The memory 801 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 801 may store a program. When the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to execute the various steps of the training method of the code completion model of the embodiments of this application.

[0175] The processor 802 may be a CPU, a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, used to execute related programs to implement the functions required by the units in the code completion model training device of the present application embodiment, or to execute the code completion model training method of the method embodiment of the present application.

[0176] The processor 802 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the training method for the code completion model of this application can be completed by the integrated logic circuits in the hardware of the processor 802 or by software instructions. The aforementioned processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an ASIC, a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory 801. The processor 802 reads the information in the memory 801 and, in conjunction with its hardware, performs the functions required by the units included in the code completion model training device of the present application embodiment, or executes the code completion model training method of the method embodiment of the present application.

[0177] The communication interface 803 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 800 and other devices or communication networks. For example, the first training data described above can be obtained through the communication interface 803.

[0178] Bus 804 may include a pathway for transmitting information between various components of device 800 (e.g., memory 801, processor 802, communication interface 803).

[0179] Figure 11 This is a schematic block diagram of the code completion device in this application. Figure 11 The code completion device 900 shown includes an acquisition unit 901 and a processing unit 902.

[0180] The acquisition unit 901 and the processing unit 902 can be used to execute the code completion method of the embodiments of this application. Specifically, the acquisition unit 901 can execute the above step 901, and the processing unit 902 can execute the above step 902.

[0181] Processing unit 902 can achieve Figure 8 The code shown completes the functionality of the neural network.

[0182] It should be understood that the processing unit 902 in the above-described device 900 may be equivalent to the processor 1002 in the device 1000 described below.

[0183] Figure 12 This is a schematic diagram of the hardware structure of the code completion device provided in the embodiments of this application. Figure 12 The code completion device 1000 shown (which can specifically be a computer device) includes a memory 1001, a processor 1002, a communication interface 1003, and a bus 1004. The memory 1001, processor 1002, and communication interface 1003 are interconnected via the bus 1004.

[0184] The memory 1001 may be a ROM, a static storage device, a dynamic storage device, or RAM. The memory 1001 may store a program. When the program stored in the memory 1001 is executed by the processor 1002, the processor 1002 and the communication interface 1003 are used to execute the various steps of the code completion method of the embodiments of this application.

[0185] The processor 1002 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the code completion device of this application embodiment, or to execute the code completion method of this application method embodiment.

[0186] The processor 1002 can also be an integrated circuit chip with signal processing capabilities. During implementation, each step of the code completion method of this application can be completed by the integrated logic circuitry in the hardware of the processor 1002 or by instructions in software form. The processor 1002 can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1001. The processor 1002 reads the information in memory 1001 and, in conjunction with its hardware, completes the functions required by the units included in the code completion device of this application embodiment, or executes the code completion method of the method embodiment of this application.

[0187] The communication interface 1003 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 1000 and other devices or communication networks. For example, the aforementioned face image can be acquired through the communication interface 1003.

[0188] Bus 1004 may include a pathway for transmitting information between various components of device 1000 (e.g., memory 1001, processor 1002, communication interface 1003).

[0189] It should be noted that, although Figure 10 The device 800 shown Figure 12 The illustrated device 1000 only shows the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, devices 800 and 1000 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 800 and 1000 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that devices 800 and 1000 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 10 , Figure 12 All the devices shown.

[0190] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different devices to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0191] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0192] In the several embodiments provided in this application, it should be understood that the disclosed systems, methods, and apparatuses can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0193] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0194] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0195] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: a Universal Serial Bus flash disk (USB flash disk, UFD), which can also be simply referred to as a USB flash drive, portable hard drive, ROM, RAM, magnetic disk, or optical disk, and other media capable of storing program code.

[0196] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A training method for a code completion model, characterized in that, include: Obtain first training data, the first training data including program code, the program code including a sequence of lexical terms represented by a two-line parallel sequence, wherein the two-line parallel sequence includes a type line and a token line, the type line being used to indicate the structured syntax information of the program code, and the token line being used to indicate the semantic information of the program code; The first training data is input into the neural network model and the neural network model is trained based on a custom Transformer multilayer neural network to obtain the target code completion model; Each node of the two-row parallel sequence includes a Type field and a Token field.

2. The training method according to claim 1, characterized in that, The step of inputting the first training data into the neural network model includes: Construct a vocabulary, which includes a Type vocabulary and a Token vocabulary. The Type vocabulary is constructed from the complete set of Type words, and the key values ​​corresponding to the indexes in the Token vocabulary are not fixed. The word sequence in the first training data is mapped to an integer sequence according to the vocabulary, and the integer sequence is input into the neural network model.

3. The training method according to claim 2, characterized in that, The morpheme sequences in the first training data include out-of-vocabulary (OOV) words, and the method further includes: The OOV words are entered sequentially from the head of the Token vocabulary list in order of their distance from the current cursor position; and / or, The OOV words are filled in sequentially from the end of the Token vocabulary list.

4. The training method according to claim 1, characterized in that, The training of the neural network model based on the custom Transformer-based multilayer neural network structure includes: The integer sequences representing Type and Token are embedded into a high-dimensional real vector sequence, and then concatenated into an 88-dimensional input vector sequence after encoding. The first transform network is used to perform feature encoding on the input vector sequence to obtain a feature map; The feature map is grammatically decoded using a second transform network, and semantically decoded using a third transform network to train the neural network. Wherein, the depth of the first transformation network is greater than the depth of the second transformation network and the depth of the third transformation network, and the depth of the third transformation network is greater than the depth of the second transformation network.

5. A method for code completion, characterized in that, include: Program code that retrieves user input; The program code is input into the target code completion model to obtain the code completion result. The target code completion model is obtained by training a neural network model using a first training data and a multi-layer neural network structure based on a modified Transformer. The first training data includes program code, which includes a sequence of lexical terms represented by a two-line parallel sequence. The two-line parallel sequence includes a type line and a token line. The type line is used to indicate the structured syntax information of the program code, and the token line is used to indicate the semantic information of the program code. Each node of the two-line parallel sequence includes a type field and a token field.

6. The method according to claim 5, characterized in that, The step of inputting the program code into the target code completion model to obtain the code completion result includes: The program code is converted into a sequence of lexical terms represented using the two-line parallel sequence; The word sequence is mapped to an integer sequence according to the vocabulary. The vocabulary includes a Type vocabulary and a Token vocabulary. The Type vocabulary is constructed from the complete set of Type vocabulary. The key value corresponding to each index in the Token vocabulary is not fixed. The code completion result is obtained based on the integer sequence.

7. The method according to claim 6, characterized in that, The step of obtaining the code completion result based on the integer sequence includes: A single-step reasoning result representing Type is obtained by performing single-step reasoning on the integer sequence representing Type; A single-step reasoning result representing the Token is obtained by performing single-step reasoning on the integer sequence representing the Token; The code completion result is obtained based on the single-step reasoning result of the representation Type and the single-step reasoning result of the representation Token.

8. The method according to claim 7, characterized in that, The method further includes: Static analysis is performed on the integer sequence to obtain the static analysis results. The step of obtaining the code completion result based on the single-step reasoning result of the representation Type and the single-step reasoning result of the representation Token includes: The code completion result is obtained by performing conditional pruning on the single-step inference results of the representation Type and the single-step inference results of the representation Token using the static analysis results.

9. The method according to any one of claims 5 to 8, characterized in that, The single-step reasoning results representing Type and the single-step reasoning results representing Token include multiple branch groups, the output results of the multiple branch groups have a similarity lower than a preset value, and the code completion results include the output results of the multiple groups.

10. A training device for a code completion model, characterized in that, include: An acquisition unit is configured to acquire first training data, the first training data including program code, the program code including a sequence of lexical terms represented by a two-line parallel sequence, wherein the two-line parallel sequence includes a type line and a token line, the type line being used to indicate the structured syntax information of the program code, and the token line being used to indicate the semantic information of the program code; each node of the two-line parallel sequence includes a type field and a token field; The training unit is used to input the first training data into the neural network model and train the neural network model based on a custom Transformer multilayer neural network to obtain the target code completion model.

11. The training device according to claim 10, characterized in that, The training unit is specifically used for: Construct a vocabulary, which includes a Type vocabulary and a Token vocabulary. The Type vocabulary is constructed from the complete set of Type words, and the key values ​​corresponding to the indexes in the Token vocabulary are not fixed. The word sequence in the first training data is mapped to an integer sequence according to the vocabulary, and the integer sequence is input into the neural network model.

12. The training device according to claim 11, characterized in that, The training unit is also used for: Enter OOV words sequentially from the head of the Token vocabulary list in order of their distance from the current cursor position; and / or The OOV words are filled in sequentially from the end of the Token vocabulary list.

13. The training device according to claim 12, characterized in that, The training unit is specifically used for: The integer sequences representing Type and Token are embedded into a high-dimensional real vector sequence, and then concatenated into an 88-dimensional input vector sequence after encoding. The first transform network is used to perform feature encoding on the input vector sequence to obtain a feature map; The feature map is grammatically decoded using a second transform network, and semantically decoded using a third transform network to train the neural network model. Wherein, the depth of the first transformation network is greater than the depth of the second transformation network and the depth of the third transformation network, and the depth of the third transformation network is greater than the depth of the second transformation network.

14. A code completion device, characterized in that, include: The acquisition unit is used to acquire program code input by the user; A processing unit is configured to input the program code into a target code completion model to obtain a code completion result. The target code completion model is obtained by training a neural network model using a first training data and a multi-layer neural network structure based on a modified Transformer. The first training data includes program code, which includes a sequence of lexical terms represented by a two-line parallel sequence. The two-line parallel sequence includes a type line and a token line. The type line is used to indicate the structured syntax information of the program code, and the token line is used to indicate the semantic information of the program code. Each node of the two-line parallel sequence includes a type field and a token field.

15. The apparatus according to claim 14, characterized in that, The processing unit is specifically used for: The program code is converted into a sequence of lexical terms represented using the two-line parallel sequence; The word sequence is mapped to an integer sequence according to the vocabulary. The vocabulary includes a Type vocabulary and a Token vocabulary. The Type vocabulary is constructed from the complete set of Type vocabulary. The key value corresponding to each index in the Token vocabulary is not fixed. The code completion result is obtained based on the integer sequence.

16. The apparatus according to claim 15, characterized in that, The processing unit is specifically used for: A single-step reasoning result representing Type is obtained by performing single-step reasoning on the integer sequence representing Type; A single-step reasoning result representing the Token is obtained by performing single-step reasoning on the integer sequence representing the Token; The code completion result is obtained based on the single-step reasoning result of the representation Type and the single-step reasoning result of the representation Token.

17. The apparatus according to claim 16, characterized in that, The processing unit is also used for: Static analysis is performed on the integer sequence to obtain the static analysis results. The step of obtaining the code completion result based on the single-step reasoning result of the representation Type and the single-step reasoning result of the representation Token includes: The code completion result is obtained by performing conditional pruning on the single-step inference results of the representation Type and the single-step inference results of the representation Token using the static analysis results.

18. The apparatus according to any one of claims 14 to 17, characterized in that, The single-step reasoning results representing Type and the single-step reasoning results representing Token include multiple branch groups, the output results of the multiple branch groups have a similarity lower than a preset value, and the code completion results include the output results of the multiple groups.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code that can be executed by a device, the program code including instructions for performing the method as described in any one of claims 1 to 4 or any one of claims 5 to 9.

20. A training device for a code completion model, characterized in that, The training device includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface to execute the training method as described in any one of claims 1 to 4.

21. A code completion device, characterized in that, The device includes a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the method as described in any one of claims 5 to 9.

22. A computer program product, characterized in that, When the computer program is executed on a computer, it causes the computer to perform the method as claimed in any one of claims 1 to 4 or any one of claims 5 to 9.

Images