Training method and system for code processing model, processing method and system for code processing model, and device and medium

Abstract

Disclosed in the present application are a training method and system for a code processing model, a processing method and system for a code processing model, and a device and a medium. The training method for a code processing model comprises: acquiring a code processing model to be trained; training the code processing model by means of a grammar and semantic training task, so as to obtain a first processing model; training the first processing model by means of a generation and retrieval training task, so as to obtain a second processing model; and performing instruction fine-tuning on the second processing model, so as to obtain a trained code processing model. The code processing model built in the method can flexibly switch between an encoder mode, a decoder mode and an encoder-decoder mode, so as to adapt to various task scenarios; in addition, a grammar and semantic training task, a generation and retrieval training task and instruction fine-tuning are used during training, so that the processing performance of the model on code data can be improved, and the application effect of the model can be improved, thereby helping to improve the user experience. The present application can be widely applied in the technical field of artificial intelligence.

Description

Code processing model training, processing methods, systems, equipment and media

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on March 26, 2024, with application number 2024103537784 and application name “Training, processing method, system, device and medium for code processing model”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence technology, and in particular to a method, system, device, and medium for training and processing a code processing model. Background Art

[0003] With the development of artificial intelligence technology, a wide range of applications have emerged across various industries. For example, in the field of software information technology, an increasing number of companies and research institutions have launched large language models for code generation to assist programmers in writing code and improve development efficiency.

[0004] The code processing models in related technologies often set their model structures according to actual business needs. For example, they are designed as encoder structures, which obtain the embedded information corresponding to the code data from large-scale training data; or they are designed as decoder structures, which are trained to have the ability to predict the next text unit, thereby realizing the function of code generation. However, in actual applications, it is found that the code processing models with encoder structures have code understanding capabilities, but almost no code generation capabilities; the code processing models with decoder structures have code generation capabilities, but weak code understanding capabilities; and the code processing models with encoder-decoder structures have both code understanding and code generation capabilities, but their capabilities are weak and their use is not flexible enough. Moreover, related technical solutions often treat code languages ​​the same as natural languages, ignoring the characteristics of the code data itself, resulting in poor model application effects and poor user experience.

[0005] In summary, the problems existing in related technologies need to be solved urgently.

[0006] Summary of the Invention

[0007] The purpose of this application is to solve one of the technical problems existing in the related art to at least a certain extent.

[0008] To this end, one purpose of embodiments of the present application is to provide training, processing methods, systems, devices and media for code processing models.

[0009] In order to achieve the above technical objectives, the technical solutions adopted in the embodiments of the present application include:

[0010] In one aspect, an embodiment of the present application provides a method for training a code processing model, the method comprising:

[0011] Get the code processing model to be trained;

[0012] Training the code processing model through grammatical and semantic training tasks to obtain a first processing model;

[0013] Training the first processing model by generating and retrieving training tasks to obtain a second processing model;

[0014] Fine-tuning the instructions of the second processing model to obtain a trained code processing model;

[0015] Among them, the code processing model includes encoder mode, decoder mode and encoder-decoder mode; in the encoder mode, each text unit can pay attention to all other text units; in the decoder mode, each text unit can pay attention to all text units before it; in the encoder-decoder mode, the text unit in the input data can pay attention to all other text units in the input data, and the text unit in the output data can pay attention to all text units before it.

[0016] In addition, the code processing model training method according to the above embodiment of the present application may also have the following additional technical features:

[0017] Furthermore, in one embodiment of the present application, the grammar and semantics training task includes a code structure type prediction task; and the training of the code processing model through the grammar and semantics training task includes:

[0018] Obtaining first training code data;

[0019] Analyzing the first training code data using a compilation tool to obtain an abstract syntax tree corresponding to the first training code data, and determining a node label corresponding to each text unit in the first training code data; the node label is used to represent a true result of a node type of a text unit in the first training code data;

[0020] In the encoder mode, inputting the first training code data into the code processing model, and predicting the node type of the text unit in the first training code data by the code processing model to obtain a first prediction result;

[0021] Determining a first loss value according to the node label and the first prediction result;

[0022] According to the first loss value, the parameters of the code processing model are updated.

[0023] Furthermore, in one embodiment of the present application, the syntax and semantics training task includes an identifier-enhanced mask prediction task; and the training of the code processing model through the syntax and semantics training task includes:

[0024] obtaining second training code data;

[0025] masking a plurality of target code segments in the second training code data to obtain third training code data; wherein the target code segments include code segments including the covering identifier;

[0026] In the encoder-decoder mode, inputting the third training code data into the code processing model, and predicting the masked code segment in the third training code data by the code processing model to obtain a second prediction result;

[0027] determining a second loss value according to the second prediction result and the target code segment;

[0028] According to the second loss value, the parameters of the code processing model are updated.

[0029] Furthermore, in one embodiment of the present application, the generation and retrieval training task includes an autoregressive code generation task; and the training of the first processing model through the generation and retrieval training task includes:

[0030] Obtaining fourth training code data;

[0031] In the decoder mode, inputting each text unit of the fourth training code data into the code processing model in sequence, and predicting the next text unit of the currently input text unit by the code processing model to obtain a third prediction result;

[0032] Determining a third loss value based on the third prediction result and each text unit of the fourth training code data;

[0033] According to the third loss value, the parameters of the code processing model are updated.

[0034] Furthermore, in one embodiment of the present application, the generation and retrieval training task includes a multimodal generation task; and the training of the first processing model through the generation and retrieval training task includes:

[0035] Obtaining fifth training code data and a target document character string corresponding to the fifth training code data;

[0036] In the encoder-decoder mode, inputting the target document string into the code processing model, and predicting corresponding code data by the code processing model to obtain a fourth prediction result;

[0037] Determining a fourth loss value based on the fourth prediction result and the fifth training code data;

[0038] In the encoder-decoder mode, inputting the fifth training code data into the code processing model, and predicting a corresponding document string by the code processing model to obtain a fifth prediction result;

[0039] Determining a fifth loss value based on the fifth prediction result and the target document string;

[0040] Update parameters of the code processing model according to the fourth loss value and the fifth loss value.

[0041] Furthermore, in one embodiment of the present application, the generation and retrieval training task includes a text retrieval code task; and the training of the first processing model through the generation and retrieval training task includes:

[0042] Acquire the search text segment and the sixth training code data;

[0043] In the encoder mode, the search text segment and the sixth training code data are input into the code processing model, and a first feature vector corresponding to the search text segment and a second feature vector corresponding to the sixth training code data are extracted by the code processing model;

[0044] Calculating a dot product of the first eigenvector and the second eigenvector;

[0045] Based on the dot product, determining a sixth loss value according to a matching relationship between the search text segment and the sixth training code data;

[0046] According to the sixth loss value, the parameters of the code processing model are updated.

[0047] Furthermore, in one embodiment of the present application, the code processing model includes an embedding layer and L Transformer layers, and the method further includes:

[0048] During the syntax and semantics training task phase, all parameters of the code processing model are updated; during the generation and retrieval training task phase, the parameters of the embedding layer and the first H Transformer layers in the code processing model are fixed, and the parameters of the last LH Transformer layers in the code processing model are updated; during the instruction fine-tuning phase, the parameters of the embedding layer, the first H Transformer layers, and the multilayer perceptron parameters of the last LH Transformer layers in the code processing model are fixed, and the self-attention parameters of the last LH Transformer layers in the code processing model are updated;

[0049] Wherein, L and H are positive integers, and L is greater than H.

[0050] On the other hand, an embodiment of the present application provides a training system for a code processing model, the system comprising:

[0051] An acquisition unit, used to acquire a code processing model to be trained;

[0052] A first training unit is configured to train the code processing model through grammar and semantic training tasks to obtain a first processing model;

[0053] A second training unit is configured to train the first processing model by generating and retrieving training tasks to obtain a second processing model;

[0054] A fine-tuning unit, configured to perform instruction fine-tuning on the second processing model to obtain a trained code processing model;

[0055] Among them, the code processing model includes encoder mode, decoder mode and encoder-decoder mode; in the encoder mode, each text unit can pay attention to all other text units; in the decoder mode, each text unit can pay attention to all text units before it; in the encoder-decoder mode, the text unit in the input data can pay attention to all other text units in the input data, and the text unit in the output data can pay attention to all text units before it.

[0056] In another aspect, an embodiment of the present application provides an electronic device, including:

[0057] at least one processor;

[0058] at least one memory for storing at least one program;

[0059] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned training method of the code processing model.

[0060] On the other hand, an embodiment of the present application also provides a computer-readable storage medium, which stores a program executable by a processor. When the program executable by the processor is executed by the processor, it is used to implement the training method of the above-mentioned code processing model.

[0061] The advantages and benefits of this application will be partially given in the following description, and partially become apparent from the following description, or learned through practice of this application:

[0062] The training, processing method, system, device and medium of the code processing model disclosed in the embodiment of the present application obtain the code processing model to be trained; train the code processing model through grammar and semantic training tasks to obtain a first processing model; train the first processing model through generation and retrieval training tasks to obtain a second processing model; perform instruction fine-tuning on the second processing model to obtain a trained code processing model. The code processing model constructed in this method can flexibly switch between encoder mode, decoder mode, and encoder-decoder mode to adapt to various task scenarios; moreover, using grammar and semantic training tasks, generation and retrieval training tasks and instruction fine-tuning to train the model during the training process can improve the model's processing performance on code data, improve the model's application effect, and help improve user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following introduction is made to the drawings of the embodiments of the present application or the related technical solutions in the prior art. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solutions of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative work.

[0064] FIG1 is a schematic diagram of an implementation environment of a code processing model training method provided in an embodiment of the present application;

[0065] FIG2 is a flow chart of a method for training a code processing model provided in an embodiment of the present application;

[0066] FIG3 is a schematic diagram showing the principle of a code processing model provided in an embodiment of the present application;

[0067] FIG4 is a schematic diagram of a flow chart of a code structure type prediction task provided in an embodiment of the present application;

[0068] FIG5 is a flowchart of an identifier-enhanced mask prediction task provided in an embodiment of the present application;

[0069] FIG6 is a schematic diagram of a process for generating an autoregressive code according to an embodiment of the present application;

[0070] FIG7 is a schematic diagram of a process flow of a multimodal generation task provided in an embodiment of the present application;

[0071] FIG8 is a flowchart of a text retrieval code task provided in an embodiment of the present application;

[0072] FIG9 is a schematic diagram of a specific application flow of a code processing model training method provided in an embodiment of the present application;

[0073] FIG10 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. The described embodiments should not be considered as limiting the present application. All other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0075] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0076] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0077] 1) Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0078] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI disciplines. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0079] 2) Large Language Models (LLMs) are deep learning-based AI models used to generate, understand, and analyze natural language text. These models typically consist of multi-layered neural networks capable of learning and predicting grammatical structure, semantic information, and contextual relationships within text. Training LLMs typically requires vast amounts of text data and computing resources to capture the various characteristics and patterns of language. Once trained, LLMs can be used for a variety of tasks, such as automatic text generation, machine translation, sentiment analysis, and question-answering systems.

[0080] With the development of artificial intelligence technology, a wide range of applications have emerged across various industries. For example, in the field of software information technology, an increasing number of companies and research institutions have launched large language models for code generation to assist programmers in writing code and improve development efficiency.

[0081] The code processing models in related technologies often set their model structures according to actual business needs. For example, they are designed as encoder structures, which obtain the embedded information corresponding to the code data from large-scale training data; or they are designed as decoder structures, which are trained to have the ability to predict the next text unit, thereby realizing the function of code generation. However, in actual applications, it is found that the code processing models with encoder structures have code understanding capabilities, but almost no code generation capabilities; the code processing models with decoder structures have code generation capabilities, but weak code understanding capabilities; and the code processing models with encoder-decoder structures have both code understanding and code generation capabilities, but their capabilities are weak and their use is not flexible enough. Moreover, existing technical solutions often treat code languages ​​the same as natural languages, ignoring the characteristics of the code data itself, resulting in poor model application effects and poor user experience.

[0082] In view of this, an embodiment of the present application provides a training method for a code processing model, in which the code processing model constructed can flexibly switch between encoder mode, decoder mode, and encoder-decoder mode to adapt to various task scenarios; moreover, during the training process, the model is trained using grammar and semantic training tasks, generation and retrieval training tasks, and instruction fine-tuning, which can improve the model's processing performance on code data, improve the model's application effect, and help improve user experience.

[0083] Please refer to Figure 1, which shows a schematic diagram of an implementation environment for a code processing model training method provided in an embodiment of the present application. In this implementation environment, the main hardware and software entities involved include a terminal device 110 and a backend server 120. The terminal device 110 and the backend server 120 are in communication with each other.

[0084] Specifically, the training method of the code processing model provided in the embodiment of the present application can be executed separately on the terminal device 110 side, or can be executed separately on the background server 120 side, or can be executed based on data interaction between the terminal device 110 and the background server 120.

[0085] For example, taking the training method of the code processing model provided in the embodiments of the present application as an example, which is based on the data interaction between the terminal device 110 and the background server 120, in some embodiments, the terminal device 110 can be used to collect relevant training data, and the background server 120 can be used to perform specific model training tasks. The user of the terminal device 110 can collect relevant training data and transmit the training data to the background server 120 by interacting with the terminal device 110. The background server 120 can use the training data to train the code processing model, and then feed back the parameters, configuration files, and other contents of the code processing model to the terminal device 110, so that the terminal device 110 can use the trained code processing model.

[0086] Among them, the terminal device 110 in the above embodiment may include a mobile phone, a computer, a smart wearable device, a PDA device, an intelligent voice interaction device, a vehicle-mounted terminal, etc., but is not limited thereto.

[0087] The backend server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0088] The terminal device 110 and the backend server 120 may establish a communication connection via a wireless network or a wired network. The wireless network or wired network uses standard communication technologies and / or protocols, and the network may be the Internet or any other network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or any combination of a virtual private network.

[0089] Of course, it can be understood that the implementation environment in Figure 1 is just some optional application scenarios of the training method of the code processing model provided in the embodiments of this application, and the actual application is not fixed to the software and hardware environment shown in Figure 1.

[0090] Below, in combination with the introduction of the aforementioned implementation environment, a training method for a code processing model provided in an embodiment of the present application is introduced and explained.

[0091] Please refer to FIG. 2 , which is a schematic diagram of a method for training a code processing model provided in an embodiment of the present application. The method for training a code processing model includes but is not limited to:

[0092] Step 210: Obtain a code processing model to be trained;

[0093] In this step, when the model training is performed, the code processing model to be trained is first obtained. In the embodiment of the present application, the code processing model can adopt a large language model (Large Language Model), which is an artificial intelligence model based on deep learning, used to process the generation, understanding and analysis tasks of natural language text. This model is usually composed of a multi-layer neural network, which can learn and predict the grammatical structure, semantic information and contextual relationships in the text. In the embodiment of the present application, there is no restriction on the tasks that the code processing model is specifically responsible for performing. For example, in some embodiments, the code processing model can be used to perform code generation tasks. For example, the user can input a piece of code data, and the code processing model can predict the code segment that may appear below based on the input code data, thereby realizing automatic generation of code. In other embodiments, the code processing model can be used to perform tasks such as code question and answer, code translation or error analysis, and this application does not limit this.

[0094] Step 220: training the code processing model through syntax and semantic training tasks to obtain a first processing model;

[0095] In this step, after obtaining the code processing model to be trained, the code processing model can be trained through grammatical and semantic training tasks. Here, a corresponding training data set needs to be prepared. This training data set can include a large number of code examples, as well as grammatical and semantic information related to these code examples. These code examples can come from open source code libraries, online resources or other available code sources, and the corresponding grammatical and semantic information can be edited by relevant automated algorithms or manually. Then, the training data can be preprocessed to convert the code examples and their corresponding grammatical and semantic information into a data format that the model can process. For example, any data format including numbers, vectors, matrices or tensors can be used, and this application is not limited to this. During the training process, the code processing model will learn how to predict its grammatical and semantic information based on the input code examples, thereby improving the code processing model's ability to understand code data. Compared with the traditional technical solution that treats code language as natural language, the application of this application can improve the performance of the code processing model, thereby improving the accuracy of subsequent tasks such as code generation or code retrieval.

[0096] In this step, the training process usually includes multiple rounds of iterations. In each round of iteration, the code processing model will update its parameters based on the training data to improve the performance of the model in grammatical and semantic prediction tasks. During the training process, some evaluation indicators can be used to evaluate the performance of the model, such as accuracy, recall rate, F1 score, etc., which are not limited in this application. After the grammatical and semantic training task is completed, a model can be obtained, which is recorded in this application as the first processing model. The first processing model is the code processing model after the grammatical and semantic training task.

[0097] Step 230: Train the first processing model by generating and retrieving training tasks to obtain a second processing model;

[0098] In this step, after obtaining the first processing model, the first processing model can be further trained based on the generation and retrieval training task, and the newly obtained model is recorded as the second processing model. Specifically, the generation and retrieval training task can include two tasks: generation tasks and retrieval tasks. The generation task is mainly used to generate specified code data or other text data; the retrieval task is mainly used to retrieve code snippets that match the input search text from the existing code library.

[0099] In this step, during the generation and retrieval training task, the model will learn how to generate specified code data based on the input data or retrieve code snippets that match the input text. This can further improve the performance of the model in code processing tasks and enhance its ability to generate code. Similarly, in this training process, multiple rounds of iterations are usually included, and in each round of iteration, the model will perform parameter updates based on the training data of the generation and retrieval tasks to improve the performance and effect of the model. In an embodiment of the present application, some evaluation indicators can be used to evaluate the performance of the model, such as the similarity between the generated code snippet and the target code, the similarity between the retrieved code snippet and the target code, etc., which are not limited in this application.

[0100] Step 240: fine-tune the instructions of the second processing model to obtain a trained code processing model;

[0101] In this step, after obtaining the second processing model, the second processing model can be fine-tuned according to the instructions, so as to obtain a trained code processing model. Specifically, in the embodiment of the present application, the goal of the instruction fine-tuning stage is to train the second processing model to adapt to specific application scenarios so that it can complete a variety of instruction tasks. For example, in some scenarios, it is hoped that the trained model can complete: ① Code question and answer: generate code segments that complete related functions based on human questions; ② Code translation: translate the code segments of language A into equivalent code in language B; ③ Code error analysis: provide analysis and solutions based on the error report. In this scenario, the natural language instructions can be aligned with the code output, and the training input is X=instruction+"\n"+input, and the output is Y=output. The encoder-decoder mode of the model is used for training, and the loss function is expressed as:

[0102] Where J6 represents the loss value corresponding to the instruction fine-tuning stage, N represents the number of training data, θ represents the parameters of the code processing model, X represents the input data of the code processing model, and Y i Indicates the expected output.

[0103] It can be understood that the training method provided in the embodiments of the present application can improve the performance of the code processing model through various types of training tasks, thereby facilitating the completion of application tasks in different scenarios, and can improve the code processing model's ability to understand, generate and retrieve code data, which is conducive to improving the user experience.

[0104] In particular, it should be noted that in the embodiment of the present application, in order to improve the adaptability of the model to various application tasks, a code processing model can be built based on the architecture of a unified language model, and the model may include an embedding layer and several Transformer layers. Specifically, the structure of the code processing model designed in the embodiment of the present application is different from the encoder structure, decoder structure or encoder-decoder structure of the existing code generation model. The structure of the code processing model in the embodiment of the present application belongs to the unified language model, and the input part that can be focused on at each position can be controlled by modifying the self-attention mask. As shown in Figure 3, it includes a total of three modes:

[0105] (a) Encoder mode: In encoder mode, each text unit can pay attention to all other text units;

[0106] (b) Decoder mode: Each text unit can pay attention to all the text units before it;

[0107] (c) Encoder-decoder mode: A text unit in the input data can pay attention to all other text units in the input data, and a text unit in the output data can pay attention to all text units before it.

[0108] Since the code processing model in the embodiment of the present application can achieve mode switching by adjusting the focus of the text unit, the same model can simultaneously complete both code understanding and code generation tasks; it can flexibly switch the encoder, decoder, and encoder-decoder modes by adjusting the mask, flexibly adapt to multi-task scenarios (code autoregressive completion, code question and answer, code retrieval, etc.), and the performance is relatively excellent.

[0109] In the embodiments of this application, existing code generation methods generally treat code as ordinary natural language for input into model training. However, this ignores the differences between code language and natural language, and ignores the rich grammatical and semantic information in code. The grammar and semantic training tasks proposed in this application use pre-training tasks proposed for code language, enabling the model to better understand the grammatical, structural, and semantic information in code segments.

[0110] Specifically, referring to FIG4 , in some embodiments, the syntax and semantics training task includes a code structure type prediction task; and the training of the code processing model through the syntax and semantics training task includes:

[0111] Obtaining first training code data;

[0112] Analyzing the first training code data using a compilation tool to obtain an abstract syntax tree corresponding to the first training code data, and determining a node label corresponding to each text unit in the first training code data; the node label is used to represent a true result of a node type of a text unit in the first training code data;

[0113] In the encoder mode, inputting the first training code data into the code processing model, and predicting the node type of the text unit in the first training code data by the code processing model to obtain a first prediction result;

[0114] Determining a first loss value according to the node label and the first prediction result;

[0115] According to the first loss value, the parameters of the code processing model are updated.

[0116] In an embodiment of the present application, the grammatical and semantic training task includes a code structure type prediction task, and the goal of the code structure type prediction task is to train the code processing model to recognize the node type in the code data. Specifically, in an embodiment of the present application, code data can be obtained in batches as training data, which is recorded here as the first training code data. Then, lexical analysis and syntax analysis can be performed by the compilation tool to obtain the abstract syntax tree (AST) of the first training code data. Subsequently, based on the abstract syntax tree, the text units corresponding to the important node types in the first training code data can be marked with node labels. For example, for example, some important node types can be selected, including nine categories: identifiers (such as function names, variable names, class names), expressions (such as arithmetic expressions, logical expressions), literals (such as integers, floating-point numbers, strings), assignment statements, conditional statements, loop statements, function calls, function parameters, and reserved words. It should be noted that a text unit can have multiple node labels, and this application does not limit this.

[0117] Next, in encoder mode, the first training code data can be input into the code processing model, and the node type of the text unit in the first training code data can be predicted by the code processing model to obtain a prediction result, which is recorded as the first prediction result in the embodiment of the present application. After obtaining the first prediction result, the prediction accuracy of the code processing model can be evaluated based on the node label and the first prediction result, thereby facilitating the iterative update of the parameters of the code processing model. Specifically, the loss value of the training can be determined based on the node label and the first prediction result. In the embodiment of the present application, it is recorded as the first loss value. After obtaining the first loss value, the code processing model can be back-propagated and trained to update the relevant parameters inside it.

[0118] Specifically, for various models in the field of artificial intelligence, the accuracy of its prediction can be measured by a loss function. The loss function is defined on a single training data and is used to measure the prediction error of a training data. Specifically, the loss value of the training data is determined by the label of the single training data and the prediction result of the model on the training data. During actual training, a training data set includes a lot of training data, so a cost function is generally used to measure the overall error of the training data set. The cost function is defined on the entire training data set and is used to calculate the average value of the prediction error of all training data, which can better measure the prediction effect of the model. For general machine learning models, based on the aforementioned cost function, plus a regularization term that measures the complexity of the model, it can be used as the objective function of training, and based on this objective function, the loss value of the entire training data set can be calculated. There are many types of commonly used loss functions, such as 0-1 loss function, square loss function, absolute loss function, logarithmic loss function, cross entropy loss function, etc., which can all be used as loss functions of machine learning models, which will not be elaborated one by one here. In the embodiment of the present application, any one of the loss functions can be selected to determine the loss value of training. For example, the loss function corresponding to the first loss value can be expressed as:

[0119] Where J0 represents the first loss value, y ^ i,j is the node label, y i,j The first prediction result.

[0120] Specifically, referring to FIG5 , in some embodiments, the syntax and semantics training task includes an identifier-enhanced mask prediction task; and the training of the code processing model through the syntax and semantics training task includes:

[0121] obtaining second training code data;

[0122] masking a plurality of target code segments in the second training code data to obtain third training code data; wherein the target code segments include code segments including the covering identifier;

[0123] In the encoder-decoder mode, inputting the third training code data into the code processing model, and predicting the masked code segment in the third training code data by the code processing model to obtain a second prediction result;

[0124] determining a second loss value according to the second prediction result and the target code segment;

[0125] According to the second loss value, the parameters of the code processing model are updated.

[0126] In an embodiment of the present application, the code processing model can also be pre-trained based on the identifier-enhanced mask prediction task. Specifically, in an embodiment of the present application, code data can be obtained in batches and recorded as the second training code data. Then, some identifiers in the second training code data can be selected, for example, 10% of the identifiers can be selected for masking, and some other code snippets can be appropriately selected for masking to obtain the third training code data. Here, the code segment covering the identifier is recorded as the target code segment. If the identifier is masked, the code semantic analysis tool is used to replace the same variable in the same scope as the identifier with the same mask symbol, and the length of the masked code segment obeys the Poisson distribution of λ=3. Then, in the encoder-decoder mode, the third training code data can be input into the code processing model, and the code processing model can be used to predict the masked code segment in the third training code data to obtain a prediction result, which is recorded as the second prediction result. Then, a loss value can be determined based on the second prediction result and the content of the target code segment, which is recorded as the second loss value. After obtaining the second loss value, the code processing model can be back-propagated and trained to update its internal related parameters. Specifically, the loss function corresponding to the second loss value can be expressed as:

[0127] Where J1 represents the second loss value, i represents the sequence number of the text unit in the target code segment, and P θ (Y i |X,Y <i ) indicates that based on the sequence part of the input data and output data Y before position i, the model predicts Y i The probability of this text unit.

[0128] Specifically, referring to FIG6 , in some embodiments, the generation and retrieval training task includes an autoregressive code generation task; and the training of the first processing model through the generation and retrieval training task includes:

[0129] Obtaining fourth training code data;

[0130] In the decoder mode, inputting each text unit of the fourth training code data into the code processing model in sequence, and predicting the next text unit of the currently input text unit by the code processing model to obtain a third prediction result;

[0131] Determining a third loss value based on the third prediction result and each text unit of the fourth training code data;

[0132] According to the third loss value, the parameters of the code processing model are updated.

[0133] In an embodiment of the present application, an autoregressive code generation task may be included in the generation and retrieval training tasks. Specifically, in an embodiment of the present application, code data can be acquired in batches and recorded as the fourth training code data. Then, in the decoder mode, each text unit of the fourth training code data can be input into the code processing model in sequence, and the code processing model can predict the next text unit of the currently input text unit to obtain the corresponding prediction result, which is recorded as the third prediction result. Then, based on the consistency of the second prediction result and the various text units of the fourth training code data, a loss value can be determined, which is recorded as the third loss value. After obtaining the third loss value, the code processing model can be back-propagated and trained to update its internal related parameters. Specifically, the loss function corresponding to the third loss value can be expressed as:

[0134] Wherein, J2 represents the third loss value, i represents the sequence number of the text unit in the fourth training code data, and P θ (X i / X ＜i ) indicates that based on the text unit before position i of the fourth training code data, the model predicts X i The probability of this text unit.

[0135] Specifically, referring to FIG. 7 , in some embodiments, the generation and retrieval training task includes a multimodal generation task; and the training of the first processing model through the generation and retrieval training task includes:

[0136] Obtaining fifth training code data and a target document character string corresponding to the fifth training code data;

[0137] In the encoder-decoder mode, inputting the target document string into the code processing model, and predicting corresponding code data by the code processing model to obtain a fourth prediction result;

[0138] Determining a fourth loss value based on the fourth prediction result and the fifth training code data;

[0139] In the encoder-decoder mode, inputting the fifth training code data into the code processing model, and predicting a corresponding document string by the code processing model to obtain a fifth prediction result;

[0140] Determining a fifth loss value based on the fifth prediction result and the target document string;

[0141] Update parameters of the code processing model according to the fourth loss value and the fifth loss value.

[0142] In an embodiment of the present application, the generation and retrieval training task may also include a multimodal generation task. For example, code data and their corresponding document strings may be obtained in batches and recorded as the fifth training code data and the target document string. Then, the input may be set to the fifth training code data, and the model may be asked to predict and output the target document string, or the input may be set to the target document string, and the model may be asked to predict and output the fifth training code data. Specifically, in an embodiment of the present application, the above-mentioned training may be implemented in the encoder-decoder mode, the target document string may be input into the code processing model, the corresponding code data may be predicted by the code processing model, and the obtained prediction result may be recorded as the fourth prediction result; the fifth training code data may be input into the code processing model, the corresponding document string may be predicted by the code processing model, and the obtained prediction result may be recorded as the fifth prediction result. Then, the fourth loss value may be determined based on the fourth prediction result and the fifth training code data, and the corresponding formula is as follows:

[0143] Where X represents the target document string, P θ (Y i |X,Y <i ) indicates that based on the sequence of input data X and output data Y before position i, the model predicts Y i The probability of this text unit.

[0144] Based on the fifth prediction result and the target document string, the fifth loss value can be determined. The corresponding formula is as follows:

[0145] Where, Y represents the fifth training code data, P θ (X i / Y,X <i ) indicates that based on the sequence part of the input data Y and the output data X before position i, the model predicts X i The probability of this text unit.

[0146] Next, according to the fourth loss value and the fifth loss value, the parameters of the code processing model can be updated.

[0147] Specifically, referring to FIG8 , in some embodiments, the generation and retrieval training task includes a text retrieval code task; and the training of the first processing model through the generation and retrieval training task includes:

[0148] Acquire the search text segment and the sixth training code data;

[0149] In the encoder mode, the search text segment and the sixth training code data are input into the code processing model, and a first feature vector corresponding to the search text segment and a second feature vector corresponding to the sixth training code data are extracted by the code processing model;

[0150] Calculating a dot product of the first eigenvector and the second eigenvector;

[0151] Based on the dot product, determining a sixth loss value according to a matching relationship between the search text segment and the sixth training code data;

[0152] According to the sixth loss value, the parameters of the code processing model are updated.

[0153] In an embodiment of the present application, the generation and retrieval training task may include a text retrieval code task. Specifically, a batch of retrieval text segments and training code data, herein referred to as the sixth training code data, may be obtained. Based on the matching relationship between the retrieval text segment and the sixth training code data, a relationship label may be determined in advance. If the sixth training code data is the code data for the retrieval text segment query, it may be determined that the two match, and the relationship label is "match." If the sixth training code data is not the code data for the retrieval text segment query, it may be determined that the two do not match, and the relationship label is "mismatch." Then, in encoder mode, the retrieval text segment and the sixth training code data may be input into a code processing model. The code processing model extracts a first feature vector corresponding to the retrieval text segment and a second feature vector corresponding to the sixth training code data. The dot product of the first feature vector and the second feature vector is then calculated. The dot product can be used to determine a matching score between the two. A larger dot product indicates a higher matching score considered by the model. Therefore, a sixth loss value may be determined based on the matching relationship between the retrieval text segment and the sixth training code data, based on the dot product. The parameters of the code processing model may also be updated based on the sixth loss value.

[0154] In some embodiments, the code processing model includes an embedding layer and L Transformer layers, and the method further includes:

[0155] During the syntax and semantics training task phase, all parameters of the code processing model are updated; during the generation and retrieval training task phase, the parameters of the embedding layer and the first H Transformer layers in the code processing model are fixed, and the parameters of the last LH Transformer layers in the code processing model are updated; during the instruction fine-tuning phase, the parameters of the embedding layer, the first H Transformer layers, and the multilayer perceptron parameters of the last LH Transformer layers in the code processing model are fixed, and the self-attention parameters of the last LH Transformer layers in the code processing model are updated;

[0156] Wherein, L and H are positive integers, and L is greater than H.

[0157] In the embodiment of the present application, when training and optimizing the model, in the first stage, all parameters are learnable when using the grammar and semantic training task for pre-training; in the second stage, the parameters of the embedding layer and the first H layers of Transformer are fixed, and only the parameters of the last LH layers of Transformer are trained; in the instruction fine-tuning stage, all parameters of the embedding layer, the first H layers of Transformer, and the MLP (multi-layer perceptron) parameters of the last LH layers of Transformer are fixed, and only the self-attention parameters of the last LH layers of Transformer are trained. In this way, the training efficiency of the model can be greatly improved, and the model can have higher processing performance.

[0158] Below, the training method of the code processing model provided in the embodiment of this application is introduced and explained in combination with specific application examples.

[0159] Referring to Figure 9, in an embodiment of the present application, when implementing the training of the code processing model, a unified language model structure can first be initialized. By modifying the self-attention mask, the input portion that can be focused on at each position can be controlled, thereby enabling the model to have multi-modal working capabilities. Then, through the first stage of pre-training tasks, the model learns the grammatical structure and semantics of the code. Through the second stage of pre-training tasks, the generation and retrieval tasks are learned. Finally, in the instruction fine-tuning stage, the model is trained to align natural language instructions with code output, enabling it to complete a variety of instruction tasks.

[0160] After completing the training of the code processing model, it can be deployed for multi-task reasoning: Since the model structure proposed in this invention uses a unified language model, it can flexibly support different downstream task scenarios. The deployment of two typical scenarios is introduced below.

[0161] Example 1: Deployment of code completion plug-in

[0162] For code completion tasks, the model needs to predict the following code segment based on any position of the user input code. Therefore, the model's decoder mode is used to predict the following several most likely text units based on the user's current input, which are used as code completion plug-ins.

[0163] Example 2: Deployment of a retrieval-enhanced code question-answering model

[0164] ① Add [CLS] to the front of each code segment X in the plug-in code library and input it into the model. Using the encoder mode, the last hidden layer vector corresponding to [CLS] is used as the vector representation E(X) of code segment X and stored in the vector database.

[0165] ② For the user's input question Q, use the method in ① to obtain the vector representation E(Q) of Q, and use the vector retrieval tool to quickly recall the vector E(X) with the highest similarity to E(Q) from the vector database.

[0166] ③ Concatenate the user input text Q and the recalled code segment X to get input = Q + "code:" + X, and input it into the model

[0167] ④Use the encoder-decoder mode to decode and obtain the output result Y.

[0168] It can be understood that in the embodiments of the present application, a unified language model structure is used to build a code processing model, which has the ability to understand and generate code. It can flexibly switch between three modes: encoder, decoder, and encoder-decoder, and complete multiple tasks such as code completion, code question and answer, and text-code retrieval. It enables the model to understand the rich grammatical, structural and semantic information in the code segment, rather than treating the code as natural language, which can improve the effect of code understanding and code generation. Moreover, through the multi-level fixed parameter low-parameter fine-tuning method, the knowledge learned in the previous stage can be retained to the maximum extent, and forgetting can be avoided. At the same time, new knowledge can be learned, which reduces the cost of training and deployment, and the recall effect is better.

[0169] The present application also provides a code processing model training system, including:

[0170] An acquisition unit, used to acquire a code processing model to be trained;

[0171] A first training unit is configured to train the code processing model through grammar and semantic training tasks to obtain a first processing model;

[0172] A second training unit is configured to train the first processing model by generating and retrieving training tasks to obtain a second processing model;

[0173] A fine-tuning unit, configured to perform instruction fine-tuning on the second processing model to obtain a trained code processing model;

[0174] Among them, the code processing model includes encoder mode, decoder mode and encoder-decoder mode; in the encoder mode, each text unit can pay attention to all other text units; in the decoder mode, each text unit can pay attention to all text units before it; in the encoder-decoder mode, the text unit in the input data can pay attention to all other text units in the input data, and the text unit in the output data can pay attention to all text units before it.

[0175] It can be understood that the contents of the above method embodiments are all applicable to the present system embodiments, the functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0176] 10 , an embodiment of the present application provides an electronic device, including:

[0177] at least one processor 1010;

[0178] at least one memory 1020, configured to store at least one program;

[0179] When at least one program is executed by at least one processor 1010, at least one processor 1010 implements the above-mentioned training method of the code processing model.

[0180] Similarly, the contents of the above method embodiments are applicable to the present electronic device embodiment. The functions specifically implemented by the present electronic device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0181] An embodiment of the present application also provides a computer-readable storage medium, which stores a program executable by the processor 1010. When the program executable by the processor 1010 is executed by the processor 1010, it is used to execute the above-mentioned training method of the code processing model.

[0182] Similarly, the contents of the above method embodiments are applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0183] In some optional embodiments, the functions / operations mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, the two boxes shown in succession may actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flow chart of the present application are provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logic flows presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0184] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present application. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the routine skills of an engineer. Therefore, a person skilled in the art can implement the present application as set forth in the claims using ordinary techniques without undue experimentation. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the appended claims and their equivalents.

[0185] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0186] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0187] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0188] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used to implement: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0189] In the above description of this specification, reference to the terms "one embodiment / example," "another embodiment / example," or "certain embodiments / examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples.

[0190] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0191] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present application, and these equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for training a code processing model, comprising: Get the code processing model to be trained; Training the code processing model through grammatical and semantic training tasks to obtain a first processing model; Training the first processing model by generating and retrieving training tasks to obtain a second processing model; Fine-tuning the instructions of the second processing model to obtain a trained code processing model; Among them, the code processing model includes encoder mode, decoder mode and encoder-decoder mode; in the encoder mode, each text unit can pay attention to all other text units; in the decoder mode, each text unit can pay attention to all text units before it; in the encoder-decoder mode, the text unit in the input data can pay attention to all other text units in the input data, and the text unit in the output data can pay attention to all text units before it.

2. A code processing model training method according to claim 1, wherein: The grammar and semantic training tasks include code structure type prediction tasks; The code processing model is trained through grammar and semantic training tasks, including: Obtaining first training code data; Analyzing the first training code data using a compilation tool to obtain an abstract syntax tree corresponding to the first training code data, and determining a node label corresponding to each text unit in the first training code data; the node label is used to represent a true result of a node type of a text unit in the first training code data; In the encoder mode, inputting the first training code data into the code processing model, and predicting the node type of the text unit in the first training code data by the code processing model to obtain a first prediction result; Determining a first loss value according to the node label and the first prediction result; According to the first loss value, the parameters of the code processing model are updated.

3. A code processing model training method according to claim 1, wherein: The grammatical and semantic training tasks include identifier-enhanced mask prediction tasks; The code processing model is trained through grammar and semantic training tasks, including: obtaining second training code data; masking a plurality of target code segments in the second training code data to obtain third training code data; wherein the target code segments include code segments including the covering identifier; In the encoder-decoder mode, inputting the third training code data into the code processing model, and predicting the masked code segment in the third training code data by the code processing model to obtain a second prediction result; determining a second loss value according to the second prediction result and the target code segment; According to the second loss value, the parameters of the code processing model are updated.

4. A code processing model training method according to claim 1, wherein: The generation and retrieval training task includes an autoregressive code generation task; The training of the first processing model by generating and retrieving training tasks includes: Obtaining fourth training code data; In the decoder mode, inputting each text unit of the fourth training code data into the code processing model in sequence, and predicting the next text unit of the currently input text unit by the code processing model to obtain a third prediction result; Determining a third loss value based on the third prediction result and each text unit of the fourth training code data; According to the third loss value, the parameters of the code processing model are updated.

5. A code processing model training method according to claim 1, wherein: The generation and retrieval training task includes a multimodal generation task; The training of the first processing model by generating and retrieving training tasks includes: Obtaining fifth training code data and a target document character string corresponding to the fifth training code data; In the encoder-decoder mode, inputting the target document string into the code processing model, and predicting corresponding code data by the code processing model to obtain a fourth prediction result; Determining a fourth loss value based on the fourth prediction result and the fifth training code data; In the encoder-decoder mode, inputting the fifth training code data into the code processing model, and predicting a corresponding document string by the code processing model to obtain a fifth prediction result; Determining a fifth loss value based on the fifth prediction result and the target document string; Update parameters of the code processing model according to the fourth loss value and the fifth loss value.

6. A code processing model training method according to claim 1, wherein: The generation and retrieval training task includes a text retrieval code task; The training of the first processing model by generating and retrieving training tasks includes: Acquire the search text segment and the sixth training code data; In the encoder mode, the search text segment and the sixth training code data are input into the code processing model, and a first feature vector corresponding to the search text segment and a second feature vector corresponding to the sixth training code data are extracted by the code processing model; Calculating a dot product of the first eigenvector and the second eigenvector; Based on the dot product, determining a sixth loss value according to a matching relationship between the search text segment and the sixth training code data; According to the sixth loss value, the parameters of the code processing model are updated.

7. A code processing model training method according to any one of claims 1 to 6, wherein: The code processing model includes an embedding layer and L Transformer layers, and the method further includes: During the syntax and semantics training task phase, all parameters of the code processing model are updated; during the generation and retrieval training task phase, the parameters of the embedding layer and the first H Transformer layers in the code processing model are fixed, and the parameters of the last LH Transformer layers in the code processing model are updated; during the instruction fine-tuning phase, the parameters of the embedding layer, the first H Transformer layers, and the multilayer perceptron parameters of the last LH Transformer layers in the code processing model are fixed, and the self-attention parameters of the last LH Transformer layers in the code processing model are updated; Wherein, L and H are positive integers, and L is greater than H.

8. A code processing model training method according to claim 1, wherein: The code processing model is trained through grammar and semantic training tasks to obtain a first processing model, including: The training is performed through multiple rounds of iterations, and in each round of iterations, the parameters of the code processing model are updated according to the training data.

9. A code processing model training system, comprising: An acquisition unit, used to acquire a code processing model to be trained; The first training unit is used to train the code processing model through grammar and semantic training tasks to obtain the first a processing model; A second training unit is configured to train the first processing model by generating and retrieving training tasks to obtain a second processing model; A fine-tuning unit, configured to perform instruction fine-tuning on the second processing model to obtain a trained code processing model; Among them, the code processing model includes encoder mode, decoder mode and encoder-decoder mode; in the encoder mode, each text unit can pay attention to all other text units; in the decoder mode, each text unit can pay attention to all text units before it; in the encoder-decoder mode, the text unit in the input data can pay attention to all other text units in the input data, and the text unit in the output data can pay attention to all text units before it.

10. An electronic device comprising: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a training method for a code processing model as described in any one of claims 1 to 8.

11. A computer-readable storage medium storing a processor-executable program, wherein the processor-executable program is used to implement a code processing model training method as described in any one of claims 1 to 8 when executed by the processor.

Images