Code generation method and related device

Abstract

Disclosed in the present application is a code generation method. The method comprises: a code development platform receiving a code generation task, and extracting code context on the basis of the code generation task, wherein the code context comprises code snippets in a project which are related to a code to be generated; the code development platform determining, on the basis of the code context and by means of a language model, whether to call an application programming interface (API) of a language service; when determining to call the API of the language service, the code development platform sending an API calling request to the language service, so as to obtain an API calling response from the language service; and on the basis of the code context and the API calling response from the language service, the code development platform performing inference by means of the language model, so as to obtain a generated code. The method endows a language model with the capability of autonomously calling an external tool, e.g., a language service, such that the language model can simulate, during inference, a human developer to use the language service to assist the language model in code generation, thereby alleviating and even eliminating the problem of hallucination caused by the language model during code generation.

Description

A code generation method and related device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on December 14, 2023, with application number 202311723682.4 and invention name “A code generation method and related equipment”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on March 15, 2024, with application number 202410305635.6 and invention name “A code generation method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of artificial intelligence (AI) technology, and in particular to a code generation method, a code development platform, a computing device cluster, a computer-readable storage medium, and a computer program product. Background Art

[0003] As software scale and complexity increase, more and more developers are exploring the use of code generation technology for software development. Code generation technology aims to reduce developers' manual programming workload and improve code development efficiency, and has attracted widespread attention from both academia and industry in software engineering (SE). In recent years, thanks to advances in artificial intelligence (AI) research in natural language processing and breakthroughs in large language models, the results of large language model-related research have enabled code generation technology to gradually move from academic research to practical application, resulting in the emergence of various intelligent programming assistant products based on large language models.

[0004] The intelligent programming assistant product mainly focuses on the text-to-code conversion (Text2Code) scenario, which is used to generate code that implements the above requirements based on the requirements described by the developer in natural language. Specifically, the developer completes the writing of code function annotations during the code writing process and triggers code generation. Then, the intelligent programming assistant product can use the generative pre-trained transformer (GPT) model to generate code snippets that implement the functions described in the annotations based on the annotations and related information provided by the developer. The intelligent programming assistant product can present the code snippets to the developer in the form of recommendations, allowing the developer to decide whether to accept or reject the recommendation, or accept it and then make further modifications.

[0005] However, in code development projects for real application scenarios, the business logic code generation effect is poor, and the generated code includes methods or variables that do not exist at all. The above non-existent methods or variables are also called hallucinations. This leads to a low code acceptance rate and makes it difficult to meet business needs.

[0006] Summary of the Invention

[0007] This application provides a code generation method that empowers a language model to autonomously call external tools such as language services. This method enables the language model to simulate human developers using language services to assist the language model in code generation during reasoning, alleviating or even eliminating the illusions caused by the language model during code generation, and improving the usability and accuracy of code generation by intelligent programming assistants in actual development scenarios. This application also provides a code development platform, computing device cluster, computer-readable storage medium, and computer program product corresponding to the above method.

[0008] In a first aspect, the present application provides a code generation method. The method can be performed by a code development platform. The code development platform can be software, which can be independently running software or integrated into other software, such as a functional module within an integrated development environment or a plug-in of an integrated development environment, or a code editor. The software can be deployed on a computing device cluster, which executes the program code of the software system, thereby executing the code generation method of the present application. In some examples, the code development platform can also be hardware, such as a computing device cluster that provides code generation capabilities, which executes the code generation method of the present application when the computing device cluster is running.

[0009] Specifically, the code development platform receives a code generation task, which is used to generate code based on the user's input information. The code development platform then extracts the code context based on the code generation task. The code context includes code snippets in the project that are related to the code to be generated. Based on the code context, the code development platform determines whether to call the application programming interface (API) of the language service through a language model. When it is determined to call the API of the language service, the code development platform can send an API call request to the language service and obtain an API call response of the language service. The language service is used to assist in code generation, and the API call response includes at least one recommended result of the language service. The code development platform obtains the generated code through language model reasoning based on the code context and the API call response of the language service.

[0010] This method gives the language model the ability to autonomously call external tools such as language services, allowing the language model to simulate human developers using language services such as the integrated development environment's automatic completion, prompt recommendations, jump viewing and other functions to assist the language model in code generation during the reasoning process, alleviating or even eliminating the hallucination problem caused by the language model during the code generation process, thereby avoiding errors in the IDE caused by the generated code, and ultimately improving the code generation usability and accuracy of the intelligent programming assistant in actual development scenarios.

[0011] In some possible implementations, the code development platform can generate code using a language model based on the code context to obtain the current result. If the current result includes an API call identifier, the API of the language service is determined to be called. This method generates code using a language model based on the code context, and then determines whether to call the language service API based on whether the current result output by the language model includes an API call identifier. This method supports code generation without API calls or code generation based on API calls, improving compatibility while ensuring the quality of code generation.

[0012] In some possible implementations, the code development platform can also use the language model to predict whether to request user assistance. User assistance refers to user participation in the code generation process, providing assistance to the language model. If so, the code development platform can receive a user response, including the user's selection of the API call response.

[0013] This method introduces user assistance, allowing users to select one or more results from the return results of the language service, simulating the process of selecting from the IDE completion list during programming by human developers, thereby achieving the effect of human-computer collaborative pair programming, thereby further improving the quality of code generation and increasing the code acceptance rate.

[0014] In some possible implementations, the code development platform can obtain the perplexity of the language model without adding the API call response and the perplexity of adding the API call response. The perplexity is used to measure how well a probability distribution or probability model predicts a sample. A probability model with low perplexity can better predict samples. Specifically in this application, the probability model can be a language model, and therefore, the perplexity can be used to measure how well the language model predicts the code. Generally speaking, the smaller the perplexity, the more accurate the language model and the better the code generation effect. When the difference in perplexity is greater than a threshold, the code development platform can determine to request user assistance.

[0015] In this way, the degree of code prediction by the language model can be measured by quantifying the perplexity, and then a decision can be made whether to request user assistance, thereby providing a gain for the language model in code generation.

[0016] In some possible implementations, the code development platform can detect whether the language model generation result includes an identifier requesting user assistance. If so, the code development platform determines to request user assistance. This method allows the code development platform to independently choose whether to request user assistance based on the language model generation result, improving compatibility and usability while ensuring code generation quality.

[0017] In some possible implementations, the code development platform can also determine at least one of the type, parameters, or timing of the language service API call based on the code context. Accordingly, the code development platform can generate an API call request based on the type or parameters of the language service API call and then send the API call request to the language service based on the timing.

[0018] In this method, the code development platform can dynamically select the type, parameters, and timing of API calls based on the code context, rather than being limited to one or two fixed language services. The language model can also independently determine the timing of calls, which is more in line with real code development scenarios and can thus improve the quality of code generation.

[0019] In some possible implementations, the code development platform can also obtain a training set, which includes training data containing API calls, including project-level code context samples, API call response samples, and predicted code samples. The code development platform can then fine-tune the base model based on the training set to obtain a language model.

[0020] This method fine-tunes the base model using training data that carries API calls, thereby aligning the training and inference processes. Compared with the method of using external intervention to influence model behavior only in the inference stage, for example, using language services / static analysis methods to try to influence the single-step sampling probability of a large model through external intervention only in the inference stage, this application adds tool call behavior to the training data during the model training stage. Through training, the language model can be made more consistent with the expected behavior, giving the language model the ability to autonomously obtain external information and reduce hallucinations.

[0021] In some possible implementations, the training set also includes training data without API calls, which includes code context samples and predicted code samples. Accordingly, the code development platform can fine-tune the base model using a mixture of training data with and without API calls to obtain a language model.

[0022] By fine-tuning using training data without API calls, the code generation capability can be maintained. By fine-tuning using training data with API calls, the API call capability and the ability to utilize external information (such as API call responses) can be given, which has higher availability.

[0023] In some possible implementations, training data is collected as follows:

[0024] The code development platform generates training data based on heuristic rules constructed based on scenarios in which human developers call language services; or,

[0025] The code development platform constructs training data based on the interactive event records of the integrated development environment IDE; or,

[0026] The code development platform constructs training data based on the user's annotation information.

[0027] The code development platform generates training data based on heuristic rules constructed based on scenarios in which human developers call language services. This enables the basic capability of building language models at a low cost. The code development platform constructs training data based on interactive event records within the integrated development environment (IDE). This improves the reliability and diversity of training data by collecting real-world human developer behavior, thereby enhancing the generalization capabilities of the language model. The code development platform constructs training data based on user-annotated code hallucinations, strengthening the language model's ability to cope with frequent hallucinations.

[0028] In some possible implementations, the language service includes an auxiliary programming service. The auxiliary programming service is used to assist in code development, such as assisting human developers in developing code, or assisting language models in generating code. For example, the language service can be a code completion service, a prompt recommendation service, or a jump view service in an IDE. This method is not limited to a specific language service type and can support calling multiple auxiliary programming services to assist the language model in generating code and improve the quality of code generation.

[0029] In some possible implementations, the code development platform can generate code through language model reasoning based on the code context and the language service API call request and response. This method constructs prompt information for the language model using a structure of code context, language service API call request, and language service API call response, mimicking the development process of human developers. This not only allows the language model to make autonomous trade-offs and comprehensively utilize contextual information and language service information, but also focuses on current key information, improving the language model's prediction quality and alleviating the problem of hallucinations.

[0030] In a second aspect, the present application provides a code development platform. The code development platform includes:

[0031] The interactive module is used to receive a code generation task, and the code generation task is used to generate code according to user input information;

[0032] A code context processing module is used to extract code context based on the code generation task. The code context includes code snippets in the project that are related to the code to be generated.

[0033] An inference module, configured to determine whether to call the language service's application programming interface (API) based on the code context and the language model;

[0034] The code context processing module is further configured to, when determining to call an API of a language service, send an API call request to the language service and obtain an API call response from the language service, wherein the language service is configured to assist in code generation, and the API call response includes at least one recommendation result of the language service;

[0035] The reasoning module is also used to obtain generated code through language model reasoning based on the code context and the API call response of the language service.

[0036] In some possible implementations, the reasoning module is specifically used to:

[0037] Based on the code context, the current result is obtained by code generation through the language model;

[0038] If the current result includes an API call identifier, determine the API that calls the language service.

[0039] In some possible implementations, the reasoning module is further configured to:

[0040] Predict whether to request user assistance through language model;

[0041] The interaction module is further configured to receive a user response if yes, where the user response includes a result of the user's selection of the API call response.

[0042] In some possible implementations, the reasoning module is specifically used to:

[0043] Get the perplexity of the language model without and with the API call response.

[0044] When the difference in perplexity is greater than a threshold, it is determined to request user assistance.

[0045] In some possible implementations, the reasoning module is specifically used to:

[0046] detecting whether a generated result of the language model includes an identifier requesting user assistance;

[0047] If so, make sure to request user assistance.

[0048] In some possible implementations, the code context processing module is further configured to:

[0049] Determine at least one of a type, parameters, or timing of a language service API call based on the code context;

[0050] The code context processing module is specifically used for:

[0051] Generate an API call request based on the type or parameters of the language service API call;

[0052] Send an API call request to the language service based on the timing.

[0053] In some possible implementations, the code development platform further includes:

[0054] The training module is used to obtain a training set, which includes training data containing API calls. The training data containing API calls includes project-level code context samples, API call response samples, and predicted code samples.

[0055] The training module is also used to fine-tune the base model according to the training set to obtain a language model.

[0056] In some possible implementations, the training set also includes training data that does not carry API calls, and the training data that does not carry API calls includes code context samples and predicted code samples;

[0057] The training module is specifically used to:

[0058] The base model is fine-tuned based on the training data with and without API calls to obtain a language model.

[0059] In some possible implementations, the training module is specifically used to:

[0060] Generate training data based on heuristic rules constructed based on scenarios where human developers call language services; or

[0061] Construct training data based on interactive event records of the integrated development environment IDE; or,

[0062] Construct training data based on the user's annotation information.

[0063] In some possible implementations, the language service includes an auxiliary programming service.

[0064] In a third aspect, the present application provides a computing device cluster. The computing device cluster includes at least one computing device, wherein the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is configured to execute instructions stored in the at least one memory, so that the computing device or computing device cluster performs the code generation method described in the first aspect or any implementation of the first aspect.

[0065] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions, wherein the instructions instruct a computing device or a computing device cluster to execute the code generation method described in the first aspect or any implementation of the first aspect.

[0066] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device or a computing device cluster, enables the computing device or computing device cluster to execute the code generation method described in the first aspect or any one of the implementations of the first aspect.

[0067] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical methods of the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments.

[0069] FIG1 is a schematic diagram of the architecture of a code development platform provided by this application;

[0070] FIG2 is a flow chart of a churn reasoning provided by this application;

[0071] FIG3 is an example diagram of a multi-task hybrid fine-tuning provided by this application;

[0072] FIG4 is a flow chart of a code generation method provided by the present application;

[0073] FIG5 is a schematic diagram of a code editing interface provided by the present application;

[0074] FIG6 is a schematic diagram of an interaction for code generation in the reasoning phase provided by this application;

[0075] FIG7 is a flowchart of a language model training method provided by the present application;

[0076] FIG8 is a schematic diagram of a code format of training data provided by this application;

[0077] FIG9 is a schematic diagram of a code format of training data including chained API calls provided by the present application;

[0078] FIG10 is a schematic diagram of the structure of a code development platform provided by this application;

[0079] FIG11 is a schematic diagram of the structure of a computing device provided by the present application;

[0080] FIG12 is a schematic diagram of the structure of a computing device cluster provided by this application;

[0081] FIG13 is a schematic diagram of the structure of another computing device cluster provided by the present application;

[0082] FIG14 is a schematic diagram of the structure of another computing device cluster provided in this application. DETAILED DESCRIPTION

[0083] The terms "first" and "second" in the embodiments of this application are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Therefore, features specified as "first" or "second" may explicitly or implicitly include one or more of the features.

[0084] First, some technical terms involved in the embodiments of this application are introduced.

[0085] Code generation refers to the process of automatically generating code based on user input, such as incomplete code or natural language descriptions, using automated tools or techniques to complete the code or implement the functionality described in the natural language description. Depending on the granularity of the generated code, code generation can include line-level code generation and method-level (or function-level) code generation.

[0086] A large language model (LLM) is a language model composed of an artificial neural network with many parameters (typically billions of weights or more) that is trained on large amounts of unlabeled text using self-supervised or semi-supervised learning. Large language models can be categorized into different types based on their architecture. Generative Pre-trained Transformer (GPT) models are widely used in code generation.

[0087] Intelligent programming assistant products based on large language models such as GPT support generating code snippets that implement the functions described in the comments based on the comments and related information provided by the developer. User feedback and large-scale survey results during the public beta phase indicate that these intelligent programming assistant products can effectively reduce the cost of developers frequently switching between actual code writing and activities such as searching for knowledge, consulting documents, and finding reusable components, thereby improving software development efficiency. However, in code development projects for real-world application scenarios, the business logic code generation effect is poor, specifically manifested in the generated code containing methods or variables that do not exist at all (commonly referred to as hallucinations in the AI ​​field).

[0088] After research, the inventors discovered that the code generation models in intelligent programming assistant products are typically trained using LLMs such as GPT. The training and reasoning methods of the code generation models are derived from natural language processing (NLP) technology, and do not fully consider the logical structure of code projects and the common call relationships between code file contents. Specifically:

[0089] During model training, LLM processes training data sequentially using natural language, which is inconsistent with the logic of actual code development. Furthermore, LLM's training method uses a text-based, context-dependent autoregressive approach, lacking the ability to introduce context into the model or invoke external tools (such as auto-completion and suggestion tools).

[0090] During the model inference phase, the context scope that the LLM based on autoregressive training can perceive is naturally limited to the text content of the current file. The generated code is likely to include calls to code that does not exist in the project, but cannot correctly call existing code in the project.

[0091] This shows that compared to other software development tools, intelligent programming assistants based on LLM are still in their infancy. Eliminating or mitigating code generation illusions requires ongoing exploration in practice. To this end, the industry has proposed various optimization solutions.

[0092] A typical optimization approach uses global context to guide the language model of the code. This approach uses monitors to guide the language model. Specifically, this approach leverages static analysis to generate code completion suggestions and constrains code generation by changing the probability distribution of the language model output. Experimental results show that this approach can significantly improve the quality of code generation.

[0093] However, this approach introduces monitors through rule definitions, supporting only a few situations or scenarios in experiments. Furthermore, due to the way data is organized and training objectives are defined during pre-training, relying solely on large models is incapable of determining whether context is sufficient and authentic. Existing methods are unable to dynamically, autonomously, and instantly incorporate external information to prevent hallucinations.

[0094] In view of this, the present application provides a code generation method. The method aims to give a language model (such as LLM) the ability to autonomously call software development tools (also collectively referred to as language services), such as the ability provided by software development tools through application programming interfaces (APIs), to alleviate the limitations of current intelligent programming assistants in code generation, and innovatively proposes a method to solve the illusion of code generation by allowing the language model to autonomously call language services. This method is inspired by the programming process of human developers and aims to simulate the process of humans writing code in an integrated development environment (IDE) (i.e., the programming site of human developers). Through training, the language model uses IDE (or other auxiliary programming services, collectively referred to as language services) to assist itself in "writing" code by calling tools such as language services. Specifically, by constructing a training set that includes language service call behaviors, tool learning (a training method that enables language models to learn to generate tool calls) can be used to train the language model's ability to autonomously call language services. During the reasoning process, the language model can simulate humans using IDE's auto-completion, prompt recommendations, jump viewing, and other functions to assist in code generation. It can even allow human developers to help the language model select from the information provided by the language service, thereby effectively alleviating the problems of low code generation practicality and acceptance rate caused by the hallucination phenomenon.

[0095] The code generation method can be performed by a code development platform. The code development platform can be software, which can be independently running software or integrated into other software, such as a functional module within an IDE or a plug-in integrated into an IDE, or a code editor. The software can be deployed on a computing device cluster, which executes the program code of the software system, thereby executing the code generation method of the present application. The software can be provided to users in the form of a software package. For example, the software can be provided to users as a new function of a client code editor or IDE with version updates, or as a new feature of a code generation plug-in based on a pre-trained language model, with version updates. Users can run the software package in a local data center or a private cloud to deploy the software. Alternatively, the software can be provided to users in the form of a cloud service, such as software as a service (SaaS). For example, the software can appear as an auxiliary coding function of a cloud code editor or development environment, exposing a functional interface to the outside in the form of a cloud service, such as an application programming interface (API), and other tools can use the above-mentioned code generation function or capability by calling the interface. In some possible implementations, the code development platform may also be hardware, such as a computing device cluster that provides code generation capabilities. When the computing device cluster is running, it executes the code generation method of the present application.

[0096] Specifically, the code development platform can receive a code generation task, and the code generation task is used to generate code according to the user's input information. The code development platform extracts the code context according to the code generation task, and the code context includes code snippets in the project related to the code to be generated. The code development platform then determines whether to call the application programming interface API of the language service based on the code context through the language model. When it is determined to call the API of the language service, the code development platform sends an API call request to the language service and obtains an API call response of the language service. Among them, the language service is used to assist in code generation, and the API call response may include at least one recommended result of the language service, such as at least one completion result of the automatic completion service, at least one prompt of the prompt recommendation service, or at least one piece of code recommended by the jump view service. The code development platform obtains the generated code through language model reasoning based on the code context and the API call response of the language service.

[0097] This method gives the language model the ability to autonomously call external tools such as language services, allowing the language model to simulate human developers using language services such as IDE's automatic completion, prompt recommendations, jump viewing and other functions to assist the language model in code generation during the reasoning process, alleviating or even eliminating the hallucination problem caused by the language model during the code generation process, thereby avoiding errors in the IDE caused by the generated code, and ultimately improving the code generation usability and accuracy of the intelligent programming assistant in actual development scenarios.

[0098] In order to make the technical solution of this application clearer and easier to understand, the system architecture of the code development platform of this application is first introduced below.

[0099] Referring to the architectural diagram of a code development platform shown in FIG1 , the code development platform 10 includes an inference platform 100, which is used to perform inference based on a trained language model to obtain generated code. Furthermore, the code development platform 10 may also include a training platform 200, which is used to fine-tune the base model based on training data carrying API calls to obtain a language model. The language model is a model for generating code, also known as a code generation model. The code generation model can be a large model such as an LLM. Based on this, the code generation model can be a large code generation model, or called a large code model or a generative code model.

[0100] The reasoning process is performed in the reasoning phase, and the training process is performed in the training phase. The reasoning process of the reasoning platform 100 and the training process of the training platform 200 are described in detail below.

[0101] The inference platform 100 is configured to receive a code generation task, which generates code based on user input information and then extracts code context based on the code generation task. The code context includes code snippets in the project related to the code to be generated. For example, the inference platform 100 may include a code context processing module, which receives the code generation task and extracts code context based on the code generation task. The inference platform 100 is also configured to determine, based on the code context, whether to call a language service API using a language model. In the example of FIG1 , the language model may be a generative code model, and the language service may be an auxiliary programming service, such as a code completion service, a prompt recommendation service, or a jump view service in an IDE, which assists in code generation. For example, the inference platform 100 may include an inference service, and the code context processing module may provide the code context to the inference service. The inference service may then call the generative code model to perform inference based on the code context to determine whether to call the auxiliary programming service API. Specifically, the generated result includes an API call identifier (token), such as ls_request or ls_req, where ls represents the language server and req represents the request, indicating that the language service API needs to be called.

[0102] The reasoning platform 100 is also used to determine the API to call the language service, send an API call request to the language service, obtain the API call response of the language service, and obtain the generated code through language model reasoning based on the code context and the API call response of the language service. The API call response includes at least one recommendation result of the language service. Further, in order to improve the performance of the language model, the reasoning platform 100 can obtain the generated code through language model reasoning based on the code context, the API call request of the language service and the API call response of the language service. As shown in Figure 1, the code context processing module in the reasoning platform 100 can send an API call request to the auxiliary programming service and obtain the API call response returned by the auxiliary programming service. The context processing module can splice the code context with the API call request and the API call response to obtain a new code context to perform a streaming reasoning loop. Specifically, the context processing module can provide the new context to the reasoning service, and the reasoning service calls the generative code model to continue code generation.

[0103] Figure 2 shows a flow chart of streaming reasoning. During the reasoning phase, the reasoning platform 100 can use streaming reasoning with real-time intervention to achieve interaction between the language model and the language service. The basic process is shown in Figure 2. Based on the code context, the reasoning service calls the generative code model to generate the current result, and then determines whether the current result includes<ls_request> If not, continue generating code. If so, call the language service and splice the language service API response into the code context to continue generating code. Repeat the above streaming reasoning until it terminates or is truncated, obtaining the final code snippet (generated code).

[0104] In some possible implementations, the reasoning platform 100 also supports human developers to intervene in code generation. Specifically, the reasoning platform 100 allows human developers (such as users) to assist language model reasoning during the reasoning process. For example, the reasoning platform 100 (or reasoning service) can monitor the changes in the perplexity (PPL) of the language model after adding the API call response (language service return result), and can generate (or insert) a token requesting user assistance when the PPL increases significantly, allowing the user to choose from the return results of the language service, simulating a process similar to that of selecting from the IDE completion list during the programming process of a human developer, thereby achieving the effect of human-computer collaborative pair programming. Among them, user assistance refers to the user's participation in the code generation process to provide assistance to the language model for code generation. For example, the user can select a recommended result from at least one recommended result output by the language service and provide it to the language model for code generation. Among them, the language service can be a code completion service, and the recommended result can be at least one code completion result recommended by the code completion service.

[0105] The key to implementing code generation in this application lies in the language model. The following introduces the process of training the language model using the training platform 200.

[0106] The training platform 200 is used to obtain a training set, which includes training data with API calls. The training data with API calls may include project-level code context samples, API call response samples, and predicted code samples, such as project-level code context samples, API call response samples, and predicted code samples extracted from a massive code repository. The base model is then fine-tuned based on the training set to obtain a language model. Fine-tuning refers to further training (or retraining) using a specific dataset based on a pre-trained model (such as a base model) to adapt the trained model to a specific task or domain. Furthermore, the training set also includes training data without API calls. The training platform 200 is used to perform mixed fine-tuning on the base model based on training data with API calls and training data without API calls to obtain a language model. Mixed fine-tuning, also known as multi-task mixed fine-tuning, refers to retraining a pre-trained model on a mixed dataset for multiple tasks (for example, a training set that is a mixture of training data with API calls and training data without API calls) to improve the performance of the trained model on all tasks.

[0107] Continuing with the example of Figure 1, the training set can be the code dataset shown in Figure 1. Training data containing API calls includes code context samples, API call response samples, and predicted code samples. Training data without API calls includes code context samples and predicted code samples. During the training phase, training platform 200 can process the code in the massive software code repository to construct training data.

[0108] In order to construct high-quality and diverse training sets to allow the language model to learn the timing and method of calling APIs, the training platform can collect training data through different methods or channels.

[0109] One way is that the training platform 200 is used to generate training data based on heuristic rules constructed based on scenarios in which human developers call language services. Among them, heuristic rules, also known as heuristic algorithms (heuristics algorithm) and heuristic rule algorithms, are a method of discovery based on empirical rules. The characteristic of heuristic rules is that when solving problems, they use past experience to select methods that have been effective, rather than systematically and in a certain way to seek answers. Heuristic rules can greatly reduce the number of attempts within a limited search space and quickly solve problems. In this embodiment, the training platform 200 can construct a heuristic rule algorithm based on scenarios in which human developers are likely to call language services, such as member references, object construction, function calls, etc., so as to achieve the basic ability to build a language model at a lower cost.

[0110] Another approach is for the training platform 200 to construct training data based on interaction event records from the integrated development environment (IDE). Specifically, the training platform 200 extracts API call behavior from the interaction event records, extracts API call response samples based on the API call behavior, and extracts context samples and predicted code samples (expected generated code) from the project's code repository. The context samples, API call response samples, and predicted code samples are then assembled into training data. By collecting the real behavior of human developers and constructing training data, the reliability and diversity of the training data can be improved, thereby improving the generalization ability of the language model.

[0111] Another approach is for the training platform 200 to construct training data based on user annotation information. The annotation information is used to mark code hallucinations. Manual annotation can enhance the language model's ability to respond to frequent hallucinations.

[0112] The training platform 200 extracts the language service call behavior through the aforementioned manual annotation, heuristic rule construction, and IDE interaction event records. It can then perform preprocessing, such as classification, extraction, screening, and deduplication, to obtain the IDE usage behavior data of human developers coding on-site. The training platform 200 can segment, encode, split, and combine the IDE usage behavior data of human developers coding on-site to construct training data. Segmentation and combination refer to classifying the data by different tasks (e.g., with and without API calls) and then splicing it into training data of a specified format.

[0113] The training platform 200 can establish an extended vocabulary to construct training data. In order to simulate the behavior of triggering language service call requests to obtain auxiliary information when human developers write code, this solution introduces a special token for marking API call requests such as<ls_request> as well as<ls_response> When the model generates this token, it starts to call the API and receive the results. Furthermore, this solution can also introduce special tokens to request user assistance in making choices, such as<user_request> as well as<user_response> , used to mark the time to request human assistance and receive the user's response. The extended vocabulary can include the above tokens. Further, the extended vocabulary can also include <req> 、 <res>, the training platform 200 can select corresponding tokens from the extended vocabulary to construct training data.

[0114] The training platform 200 generates an API call request based on the API call token and adds the API call request and the language service's response to the API call request (i.e., the API call response) to the training data. Furthermore, the API call supports chained API calls by the language model, which allows the language model to continue issuing API call requests after receiving a response, simulating the multiple calls humans make when faced with complex scenarios.

[0115] The training platform 200 can use a multi-task hybrid fine-tuning method to allow the language model to maintain its normal code generation capabilities while giving it the ability to call language services and utilize external information. For example, the training platform can use three types of task hybrid fine-tuning. The three types of tasks may include:

[0116] a) Ordinary code generation that does not involve API calls (given the code context, predict the generated code);

[0117] b) API call request generation (given code context, predict API calls);

[0118] c) Code generation based on API call responses (return results) (e.g., given a code context and an API call request and an API call response, predicting and generating code).

[0119] Figure 3 shows an example of multi-task hybrid fine-tuning. The multi-tasks include Task 1, Task 2, and Task 3. Task 1 generates code without API calls, Task 2 generates API calls, and Task 3 generates code with API calls. Fine-tuning Task 1 maintains code generation capabilities, fine-tuning Task 2 adds API call capabilities, and fine-tuning Task 3 enables the utilization of external information (such as API call responses).

[0120] This aligns the training and reasoning processes, building an intelligent framework with a large code model as the dispatching center. During the training phase, high-quality, diverse training data is constructed to empower the language model with the ability to call language services, allowing the language model to serve as the brain and core for dispatching, the language service as a tool aid, and human developers as pair programmers. Compared to methods that only use external intervention to influence model behavior during the reasoning phase, such as attempting to influence the single-step sampling probability of a large model through external intervention using language services or static analysis methods, this solution incorporates tool call behavior into the training data during the model training phase. Through training, the language model can be made more consistent with expected behavior, simulating the prompts humans receive from an IDE, giving the language model the ability to autonomously acquire external information and reduce hallucinations.

[0121] In this application, the language model is given the ability to autonomously, dynamically, and instantly access external contextual information by calling a language service. During the inference process, the language model can autonomously generate a language service API call token, dynamically select the type, parameters, or timing of the language service API call based on the current code context, and then issue a language service call request. The language service API call request can be executed by an external program, and the execution results returned by the external program can be appended to the current context (such as the current code context) to continue the language model's inference.

[0122] Compared to isolating information acquisition from the language model, this solution allows the language model to independently choose when to call external tools like language services and independently obtain external information relevant to the current generation. This approach is more aligned with the background knowledge required by human developers in real-world programming and is particularly important for object-oriented code generation. Furthermore, compared to approaches where external information only provides a macro-level reference or intervention in the language model's behavior, this solution embeds the results of static analysis into the language model's code context, allowing the model to independently choose how to utilize external information.

[0123] Compared to fully automated code generation, this solution also allows optional human intervention, that is, when the model has difficulty determining how to use external information, humans can assist the model in making choices based on their own knowledge and abilities.

[0124] Although code generation technology has made great progress in the past two years, it still cannot replace human developers in the foreseeable future. The global perspective, domain knowledge, association and reasoning ability of the latter are difficult for current artificial intelligence models to possess, and the scenarios where related technologies perform better are still relatively common simple programming tasks. Therefore, the current positioning of related products is increasingly tending to be an assistant that provides auxiliary functions rather than a robot for fully automatic programming. In this context, the significance of this application lies in combining the respective advantages of software analysis tools and artificial intelligence models, maximizing the empowerment and gains that code generation tools bring to developers, and helping developers complete complex tasks with high efficiency and high-quality development.

[0125] Based on the aforementioned code development platform 10, the present application further provides a code generation method. The code generation method of the present application is described in detail below with reference to the accompanying drawings.

[0126] Referring to the flowchart of a code generation method shown in FIG4 , the method includes the following steps:

[0127] S402: The code development platform 10 receives a code generation task.

[0128] The code generation task is used to generate code based on the user's input information. The code generation task can be a task triggered in a software project (Software Project) for generating code. A software project, also referred to as a project, is an engineering file created by a developer for the software to be developed during software development. A project can include multiple code files, such as code files used to implement different functions or features of the software. The code files in a project can be developed independently by one user or collaboratively by multiple users. During development, users can use the code generation capability to automatically generate code.

[0129] The user's input information may include at least one of the task description information or input code for the code generation task. The task description information may be a task description in natural language. For example, when the code generation task is used to generate code for a target method or target function, the task description information may be a description of the requirements for creating the target method or target function, and the task description information may typically be input in the form of comments. The input code may include a function declaration or a method declaration. Taking a function declaration as an example, a function declaration may include a function name and parameter names. Accordingly, the code generation task may be a code snippet that generates a function body.

[0130] Specifically, the code development platform 10 may present a code editing interface to the user, wherein the code editing interface may be a user interface of a code editor, and the code editing interface may be a graphical user interface (GUI) or a command user interface (CUI).

[0131] For ease of description, this application uses a code editing interface as a GUI example. As shown in Figure 5, the code editing interface 500 can include an editing window 502 for a code file. The user can enter task description information 504 in the editing window 502. The task description information 504 is used to describe the code to be generated. In this example, the task description information 504 can be "Create a http sever instance and start it", as well as input method declaration 506, such as "public void init(){}". The user can trigger the code generation control 508 of the code editing interface 500, thereby triggering the code generation operation. Accordingly, the code development platform 10 can receive the task description information 504 and method declaration 506 (input code) input by the user, and then generate code based on the task description information 504 and method declaration 506.

[0132] It should be noted that FIG5 is only an example of the user's input information including task description information and input code. In other possible implementation methods of the embodiment of the present application, the user's input information may also include task description information, or include input code. For example, when the user is editing the code of some methods, the user can directly write the method without adding task description information in the form of comments. For another example, when the user is editing the code of some methods, the user can add task description information in the form of comments without entering the code. Among them, when entering the task description information 504 in the form of comments, the user can first enter the keyword of the comment, such as the " / * / " character, the @ character or the # character, and then enter the task description information. This can avoid executing the above-mentioned task description information 504 when executing the code file.

[0133] 5 illustrates an example of a user triggering a code generation control 508 to trigger a code generation operation. In actual applications, the code development platform 10 also supports triggering a code generation operation in other ways. For example, the code development platform 10 also supports triggering a code generation operation using a shortcut key or menu (such as a right-click menu).

[0134] S404 : The code development platform 10 extracts a code context according to the code generation task.

[0135] The code context includes code snippets in the project related to the code to be generated. This code context is project-level, encompassing not only the context within the current code file (also known as the in-file context) but also the cross-file context. Cross-file context can include the context of other code files in the project. Other code files in the project can be code files referenced or imported by the current code file.

[0136] S406: The code development platform 10 determines whether to call the language service API based on the code context and the language model. If so, execute S408.

[0137] Specifically, the code development platform 10 can generate code using a language model based on the code context to obtain the current result. If the current result includes an API call identifier, for example,<ls_request> or <request> 、 <req>, the code development platform 10 determines to call the API of the language service.

[0138] S408 : The code development platform 10 sends an API call request to the language service and obtains an API call response from the language service.

[0139] The code development platform 10 may determine at least one of the type, parameters, or timing of the language service API call based on the code context. Accordingly, the code development platform 10 may generate an API call request based on the type or parameters of the language service API call. For example, the code development platform 10 may assemble the API call parameters based on a corresponding type of API call template to generate an API call request. The code development platform 10 may send the API call request to the language service based on the timing determined based on the code context.

[0140] S410 , the code development platform 10 obtains generated code through language model reasoning based on the code context and the API call response of the language service.

[0141] The code development platform 10 can splice the code context and the API call response of the language service, input the splicing result (new code context) into the language model, and obtain the generated code through language model reasoning. The language model can be a large code model, and the code development platform 10 can input the splicing result into the language model in the form of a prompt. The language model can perform reasoning based on the prompt, and the code development platform 10 can obtain the generated code reasoned by the language model. Furthermore, in order to enable the language model to obtain richer information, the code development platform 10 can also add the API call request of the language service to the splicing. For example, the code development platform 10 can splice the code context with the API call request and the API call response of the language service, and input the splicing result into the language model in the form of a prompt for reasoning to obtain the generated code. When the code development platform 10 splices the code context, the API call request and the API call response of the language service to generate the prompt, it can also add the code context, API call request and API call response to the specified location of the prompt template, thereby providing different types of information to the language model display.

[0142] In some possible implementations, the code development platform 10 may also determine whether to request user assistance through a language model. If so, the code development platform 10 may also receive a user response, including the user's selection of an API call response. The code development platform 10 may concatenate the API call request, the user's selection of the API call response, and the code context, input the concatenation result into the language model, and generate code through language model inference.

[0143] Among them, the code development platform 10 obtains the perplexity of the language model without adding the API call response and the perplexity of adding the API call response. Perplexity is used to measure how well a probability distribution or probability model predicts a sample. A probability model with low perplexity can better predict samples. Specifically in this application, the probability model can be a language model, and therefore, perplexity can be used to measure how well the language model predicts code. A language model can be regarded as a probability distribution over an entire sentence or paragraph. The language model can estimate the probability of a line of code or a section of code appearing based on each word (token), and can be normalized using the length of a line of code or a section of code (the number of tokens included in a line of code or a section of code). In some examples, perplexity can also be characterized by the average branch factor, which indicates how many options there are when predicting the next token. Generally speaking, the smaller the perplexity, the more accurate the language model and the better the code generation effect. When the perplexity difference is greater than a threshold, for example, when the perplexity difference between the language model without the API call response and the language model with the API call response is greater than the threshold, the code development platform 10 determines to request user assistance.

[0144] The code development platform 10 can detect whether the generated result of the language model includes an identifier requesting user assistance. Specifically, the code development platform 10 can generate an identifier requesting user assistance when the difference between the perplexity of the language model without the API call response and the perplexity of the language model with the API call response is greater than a threshold, for example,<user_request> , the code development platform 10 can detect whether the generated result includes<user_request> If so, the code development platform 10 may determine to request user assistance.

[0145] Based on the above description, the code generation method of the present application supports the language model to autonomously generate API call tokens during the reasoning process and issue API call requests for language services. It can dynamically select the type, parameters, and timing of API calls based on the code context, and is not limited to one or two fixed language services. The language model can also independently determine the timing of the call, which is more in line with the actual code development scenario. Moreover, this method constructs the context input to the language model with the structure of code context, API call requests for language services, and API call responses for language services, simulating the development process of human developers. It not only allows the model to make autonomous trade-offs and comprehensively utilize project-level code context and language service information, but also focuses on current key information, improves the prediction quality of the language model, and alleviates the hallucination problem.

[0146] Next, the possible implementation scheme of this application in the reasoning stage will be explained from the perspective of human-computer interaction.

[0147] Like other similar tools, this application is primarily implemented as an extension or plug-in for a code editor or IDE. Therefore, the front-end interface is embedded in the IDE as auxiliary programming tools (also known as auxiliary coding tools) such as code generation and completion. Currently, many code generation plug-in tools provide a similar experience. Unlike such tools, this solution provides the following options: mark the call location of external tools such as language services. Clicking this option will display the specific information of the language service API call, including the called API name, parameters, and return content, as shown in Figure 6.

[0148] As shown in Figure 6, the user triggers the code generation task in the code editor or IDE code editing interface, and the language model can be generated at MatcherUtils.<ls_request> , the language model detects<ls_request> , you can call the language service API, and then the language service returns the API call response, specifically LSP::getCompletion(). The language model continues to generate code based on the spliced ​​code context and API call request and API call response, repeating the above process until a terminator is generated or truncation is generated, and the code generation task is completed.

[0149] The above introduces the reasoning process of the language model. The following introduces the training process of the language model.

[0150] Referring to FIG7 , a flow chart of a language model training process is shown, which specifically includes the following steps:

[0151] S702: The code development platform 10 obtains a training set.

[0152] The training set includes training data with API calls. In order to give the language model the ability to autonomously call the language service API and interact with external tools such as the language service in real time, this application introduces a special token for marking API call requests and API call responses, such as<ls_request> and< / ls_request> As an API call request token,<ls_response> and< / ls_response> The API call response token is then added to the code context to obtain training data containing the API call. The language service can be an auxiliary programming service, such as an IDE's code generation service or code completion service.

[0153] For ease of understanding, this application also provides an example of training data that carries API calls. Figure 8 shows the code format of a piece of training data, which includes code context, API call request, and API call response, where the code context includes the code in Figure 8.<ls_request> In the previous code snippet, the API call request includes<ls_request> The following code snippets, such as LSP::getCompletion(), API call responses include<ls_response> The following code snippet shows the API call response that can be inserted into the API call request.

[0154] Furthermore, the API call form design allows the language model to chain call APIs to simulate human behavior when facing complex scenarios. The code format of chain call APIs can be found in Figure 9. For example, for an API call request, after inserting the API call response, you can continue to initiate an API call request and insert the API call response corresponding to the API call request. In the example of Figure 9, the API call response of LSP::getCompletion() is inserted into the training data, such as parseURIPatternLable, parseURIPatternTemplate, and parseURIRefAttrInConfig. The following API call request is also inserted: LSP::getDef("parseURIRefAttrInConfig") and the API call response corresponding to the API call request.

[0155] In order to construct a high-quality and diverse training set so that the language model can learn the timing and method of calling the API, the code development platform 10 can collect training data through different methods or channels. Specifically, the code development platform 10 can generate training data based on heuristic rules constructed based on scenarios in which human developers call language services. Based on scenarios in which human developers are likely to call language services, such as member references, object construction, function calls, etc., heuristic rule algorithms are constructed to achieve the basic ability to build a language model at a lower cost. Alternatively, the code development platform 10 can construct training data based on the interaction event records of the integrated development environment IDE. By collecting the real behavior of human developers, the code development platform 10 can improve the reliability and diversity of training data to improve the generalization ability of the language model. The code development platform 10 can also construct training data based on the user's annotation information. The annotation information is used to mark the occurrence of code hallucinations. Through manual annotation, the response ability of the language model in scenarios where hallucinations occur frequently can be enhanced.

[0156] S704. The code development platform 10 fine-tunes the base model according to the training set to obtain a language model.

[0157] The code development platform 10 can fine-tune the base model through tool learning based on the training set to obtain a language model. The base model can be a pre-trained model, such as a pre-trained LLM. Tool learning can be mainly divided into two categories: tool-augmented learning and tool-oriented learning. The core difference lies in whether the base model is enhanced through tool execution during the learning process (tools serve AI), or the use of tools is optimized through the base model (AI serves the tool).

[0158] This example uses tool-enhanced learning (TAL) to enhance the performance of the base model. TAL leverages the execution results of various external tools to enhance the performance of the base model. In this paradigm, the execution results of external tools, such as language services, are considered external information to aid in generating high-quality output. The return results or output of external tools, such as language services, are used as additional information to assist in code generation for the language model, addressing the issue of factual inaccuracies during language model generation, particularly during LLM generation.

[0159] In some possible implementations, the training set also includes training data that does not include API calls. This training data includes code context samples and predicted code samples. Accordingly, the code development platform 10 can perform mixed fine-tuning on the base model based on the training data that includes and does not include API calls to obtain a language model.

[0160] It should be noted that the embodiment of FIG. 7 can be implemented independently or combined with the aforementioned embodiments, and this application does not impose any limitation on this.

[0161] During the training phase, this method builds high-quality and diverse training data, giving the language model the ability to call language services. The language model serves as the core for command and dispatch, and the language service serves as an auxiliary tool. This allows the training and reasoning processes to be aligned, and an intelligent framework with the language model as the dispatching center to be constructed.

[0162] Based on the aforementioned code generation method, the present application further provides a code development platform 10. As shown in FIG10 , the code development platform 10 includes:

[0163] Interaction module 1002, for receiving a code generation task, where the code generation task is for generating code based on user input information;

[0164] A code context processing module 1004 is configured to extract a code context according to a code generation task, wherein the code context includes code snippets in the project related to the code to be generated;

[0165] The reasoning module 1006 is used to determine whether to call the application programming interface (API) of the language service according to the code context and the language model;

[0166] The code context processing module 1004 is further configured to, when determining to call an API of a language service, send an API call request to the language service and obtain an API call response from the language service, wherein the language service is configured to assist in code generation and the API call response includes at least one recommendation result of the language service;

[0167] The reasoning module 1006 is further configured to obtain generated code through language model reasoning based on the code context and the API call response of the language service.

[0168] The interaction module 1002, the code context processing module 1004, and the reasoning module 1006 may be modules in the reasoning platform 100 in the embodiment of FIG1. ​​The interaction module 1002 is not shown in FIG1, the code context processing module 1004 may be the module that performs code context processing in FIG1, and the reasoning module 1006 may be the module that performs reasoning in FIG1, which may be implemented by calling a language model.

[0169] Exemplarily, the interaction module 1002 , the code context processing module 1004 , and the reasoning module 1006 may be implemented by hardware or software.

[0170] When implemented via software, the interaction module 1002, code context processing module 1004, and inference module 1006 can be applications running on a computing device, such as a computing engine. These applications can be provided to users via virtualization services. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, and container services. VM services can use virtualization technology to create a virtual machine (VM) resource pool across multiple physical hosts, providing users with VMs on demand. BMS services create a virtual BMS resource pool across multiple physical hosts, providing users with BMSs on demand. Container services create a virtual container resource pool across multiple physical hosts, providing users with containers on demand. A VM is a simulated virtual computer, or logically a single computer. BMS is a scalable, high-performance computing service with computing performance comparable to traditional physical machines and secure physical isolation. Containers are a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service and container service in the above-mentioned virtualization services are only specific examples. In actual applications, virtualization services can also be other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0171] When implemented through hardware, the interaction module 1002, the code context processing module 1004, and the reasoning module 1006 may include at least one computing device, such as a server. Alternatively, the interaction module 1002, the code context processing module 1004, and the reasoning module 1006 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0172] In some possible implementations, the reasoning module 1006 is specifically configured to:

[0173] Based on the code context, the current result is obtained by code generation through the language model;

[0174] If the current result includes an API call identifier, determine the API that calls the language service.

[0175] In some possible implementations, the reasoning module 1006 is further configured to:

[0176] Predict whether to request user assistance through language model;

[0177] The interaction module 1002 is further configured to receive a user response if yes, where the user response includes a selection result of the user for the API call response.

[0178] In some possible implementations, the reasoning module 1006 is specifically configured to:

[0179] Get the perplexity of the language model without and with the API call response.

[0180] When the difference in perplexity is greater than a threshold, it is determined to request user assistance.

[0181] In some possible implementations, the reasoning module 1006 is specifically configured to:

[0182] detecting whether a generated result of the language model includes an identifier requesting user assistance;

[0183] If so, make sure to request user assistance.

[0184] In some possible implementations, the code context processing module 1004 is further configured to:

[0185] Determine at least one of a type, parameters, or timing of a language service API call based on the code context;

[0186] The code context processing module 1004 is specifically used to:

[0187] Generate an API call request based on the type or parameters of the language service API call;

[0188] Send an API call request to the language service based on the timing.

[0189] In some possible implementations, the code development platform 10 further includes:

[0190] A training module 1008 is configured to obtain a training set, wherein the training set includes training data containing API calls, wherein the training data containing API calls includes project-level code context samples, API call response samples, and predicted code samples;

[0191] The training module 1008 is also used to fine-tune the base model according to the training set to obtain a language model.

[0192] The training module 1008 may be implemented by hardware or software.

[0193] When implemented via software, the training module 1008 may be an application running on a computing device, such as a computing engine. This application may be provided to users via virtualization services, such as VM services, BMS services, or container services. When implemented via hardware, the training module 1008 may include at least one computing device, such as a server. Alternatively, the training module 1008 may be implemented using an ASIC or a PLD.

[0194] In some possible implementations, the training set also includes training data that does not carry API calls, and the training data that does not carry API calls includes code context samples and predicted code samples;

[0195] The training module 1008 is specifically used for:

[0196] The base model is fine-tuned based on the training data with and without API calls to obtain a language model.

[0197] In some possible implementations, the training module 1008 is specifically configured to:

[0198] Generate training data based on heuristic rules constructed based on scenarios where human developers call language services; or

[0199] Construct training data based on interactive event records of the integrated development environment IDE; or,

[0200] Construct training data based on the user's annotation information.

[0201] In some possible implementations, the language service includes an auxiliary programming service.

[0202] This application also provides a computing device 1100. As shown in Figure 11, computing device 1100 includes a bus 1102, a processor 1104, a memory 1106, and a communication interface 1108. Processor 1104, memory 1106, and communication interface 1108 communicate with each other via bus 1102. Computing device 1100 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 1100.

[0203] Bus 1102 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG11 illustrates a single bus line, but this does not imply a single bus or type of bus. Bus 1102 may include a path for transmitting information between various components of computing device 1100 (e.g., memory 1106, processor 1104, and communication interface 1108).

[0204] The processor 1104 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0205] The memory 1106 may include a volatile memory, such as a random access memory (RAM). The memory 1106 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD). The memory 1106 stores executable program code, and the processor 1104 executes the executable program code to implement the aforementioned code generation method. Specifically, the memory 1106 stores instructions for the code development platform 10 to execute the code generation method.

[0206] The communication interface 1108 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 1100 and other devices or a communication network.

[0207] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0208] As shown in Figure 12, the computing device cluster includes at least one computing device 1100. The memory 1106 of one or more computing devices 1100 in the computing device cluster may store instructions for executing the code generation method on the same code development platform 10.

[0209] In some possible implementations, one or more computing devices 1100 in the computing device cluster may also be used to execute some of the instructions for executing the code generation method on the code development platform 10. In other words, a combination of one or more computing devices 1100 may jointly execute the instructions for executing the code generation method on the code development platform 10.

[0210] It should be noted that the memory 1106 in different computing devices 1100 in the computing device cluster may store different instructions for executing part of the functions of the code development platform 10 .

[0211] FIG13 illustrates a possible implementation. As shown in FIG13 , two computing devices 1100A and 1100B are connected via a communication interface 1108. The memory of computing device 1100A stores instructions for executing the functions of interaction module 1002 and code context processing module 1004. The memory of computing device 1100B stores instructions for executing the functions of reasoning module 1006. Furthermore, the memory of computing device 1100B may also store instructions for executing the functions of training module 1008. In other words, the memories 1106 of computing devices 1100A and 1100B jointly store instructions for the code open platform 10 to execute the code generation method.

[0212] The connection method between the computing device clusters shown in Figure 13 can be considered to be based on the fact that the code generation method provided in this application requires a large amount of computing resources for model reasoning or model training. Therefore, it is considered to delegate the functions implemented by the reasoning module 1006 and the training module 1008 to the computing device 1100B.

[0213] It should be understood that the functionality of the computing device 1100A shown in FIG13 may also be implemented by multiple computing devices 1100. Similarly, the functionality of the computing device 1100B may also be implemented by multiple computing devices 1100.

[0214] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network or a local area network, etc. FIG14 shows a possible implementation. As shown in FIG14 , two computing devices 1100C and 1100D are connected via a network. Specifically, the connection to the network is made through a communication interface in each computing device. In this type of possible implementation, the memory 1106 in the computing device 1100C stores instructions for executing the functions of the interaction module 1002 and the code context processing module 1004. At the same time, the memory 1106 in the computing device 1100D stores instructions for executing the functions of the reasoning module 1006 and the training module 1008.

[0215] The connection method between the computing device clusters shown in Figure 14 can be considered to be that the code generation method provided in this application requires more computing power resources for model reasoning or model training, so it is considered to entrust the functions implemented by the reasoning module 1006 and the training module 1008 to the computing device 1100D for execution.

[0216] It should be understood that the functionality of the computing device 1100C shown in FIG14 may also be implemented by multiple computing devices 1100. Similarly, the functionality of the computing device 1100D may also be implemented by multiple computing devices 1100.

[0217] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned code development platform 10 for executing the code generation method.

[0218] The present application also provides a computer program product comprising instructions. The computer program product may be software or a program product comprising instructions that can be run on a computing device or stored in any available medium. When the computer program product is run on at least one computing device, the at least one computing device executes the aforementioned code generation method.

[0219] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the protection scope of the technical solutions of the various embodiments of the present invention.< / req> < / request> < / res> < / req>

Claims

1. A code generation method, characterized in that: The method comprises: The code development platform receives a code generation task, wherein the code generation task is used to generate code according to user input information; The code development platform extracts a code context according to the code generation task, wherein the code context includes code snippets related to the code to be generated in the project; The code development platform determines whether to call an application programming interface API of a language service through a language model according to the code context; When it is determined to call the API of the language service, the code development platform sends an API call request to the language service, obtains an API call response of the language service, the language service is used to assist in code generation, and the API call response includes at least one recommendation result of the language service; The code development platform obtains generated code through reasoning through the language model according to the code context and the API call response of the language service.

2. The method according to claim 1, characterized in that The code development platform determines whether to call an application programming interface API of a language service according to the code context by using a language model, including: The code development platform generates code through a language model according to the code context to obtain a current result; If the current result includes an API call identifier, determine the API that calls the language service.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The code development platform predicts whether to request user assistance through the language model; If so, the code development platform receives a response from the user, where the response from the user includes a selection result of the user for the API call response.

4. The method according to claim 3, characterized in that The code development platform determines whether to request user assistance through the language model, including: The code development platform obtains the perplexity of the language model before and after the API call response is added; When the difference in the perplexity is greater than a threshold, the code development platform determines to request user assistance.

5. The method according to claim 3, characterized in that: The code development platform determines whether to request user assistance through the language model, including: The code development platform detects whether the generation result of the language model includes an identifier requesting user assistance; If so, the code development platform determines to request assistance from the user.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The code development platform determines at least one of a type, a parameter, or a timing of an API call of the language service according to the code context; The code development platform sends an API call request to the language service, including: The code development platform generates an API call request according to the type or parameter of the API call of the language service; The code development platform sends the API call request to the language service according to the timing.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: The code development platform obtains a training set, wherein the training set includes training data carrying API calls, and the training data carrying API calls includes project-level code context samples, API call response samples, and predicted code samples; The code development platform fine-tunes the base model according to the training set to obtain the language model.

8. The method according to claim 7, characterized in that The training set also includes training data without API calls, and the training data without API calls includes code context samples and predicted code samples; The code development platform fine-tunes the base model according to the training set to obtain the language model, including: The code development platform performs mixed fine-tuning on the base model according to the training data carrying the API call and the training data not carrying the API call to obtain the language model.

9. The method according to claim 7 or 8, characterized in that: The training data is collected in the following way: The code development platform generates the training data according to heuristic rules constructed based on scenarios in which human developers call language services; or, The code development platform constructs the training data according to the interactive event records of the integrated development environment IDE; or, The code development platform constructs the training data according to the user's annotation information.

10. The method according to any one of claims 1 to 9, characterized in that: The language services include auxiliary programming services.

11. A code development platform, characterized in that: The code development platform includes: An interaction module, used for receiving a code generation task, wherein the code generation task is used for performing code generation according to user input information; A code context processing module, used to extract code context according to the code generation task, wherein the code context includes code snippets related to the code to be generated in the project; An inference module, configured to determine whether to call an application programming interface API of a language service according to the code context through a language model; The code context processing module is further configured to, when determining to call the API of the language service, send an API call request to the language service, and obtain an API call response of the language service, wherein the language service is used to assist in code generation, and the API call response includes at least one recommendation result of the language service; The reasoning module is further used to obtain generated code through reasoning through the language model according to the code context and the API call response of the language service.

12. The code development platform according to claim 11, characterized in that: The reasoning module is specifically used for: According to the code context, code generation is performed through a language model to obtain a current result; If the current result includes an API call identifier, determine the API that calls the language service.

13. The code development platform according to claim 11 or 12, characterized in that: The reasoning module is also used to: Predicting whether to request user assistance through the language model; The interaction module is further configured to receive a response from the user, where the response includes a selection result of the user to the API call response.

14. The code development platform according to claim 13, characterized in that: The reasoning module is specifically used for: Obtaining the perplexity of the language model without adding the API call response and the perplexity of the language model with adding the API call response; When the difference in the perplexity is greater than a threshold, it is determined to request user assistance.

15. The code development platform according to claim 13, characterized in that: The reasoning module is specifically used for: detecting whether the generated result of the language model includes an identifier requesting user assistance; If so, determine to request the user for assistance.

16. The code development platform according to any one of claims 11 to 15, characterized in that: The code context processing module is also used for: Determine at least one of a type, a parameter, or a timing of an API call of the language service according to the code context; The code context processing module is specifically used for: Generate an API call request according to the type or parameter of the API call of the language service; According to the timing, the API call request is sent to the language service.

17. The code development platform according to any one of claims 11 to 16, characterized in that: The code development platform also includes: A training module, used to obtain a training set, wherein the training set includes training data carrying API calls, wherein the training data carrying API calls includes project-level code context samples, API call response samples, and predicted code samples; The training module is also used to fine-tune the base model according to the training set to obtain the language model.

18. The code development platform according to claim 17, characterized in that: The training set also includes training data without API calls, and the training data without API calls includes code context samples and predicted code samples; The training module is specifically used for: The base model is mixed and fine-tuned according to the training data carrying the API call and the training data not carrying the API call to obtain the language model.

19. The code development platform according to claim 17 or 18, characterized in that: The training module is specifically used for: Generate the training data according to heuristic rules constructed based on scenarios in which human developers call language services; or, Constructing the training data according to the interactive event records of the integrated development environment IDE; or, The training data is constructed according to the user's annotation information.

20. The code development platform according to any one of claims 11 to 19, characterized in that: The language services include auxiliary programming services.

21. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory, wherein the at least one memory stores computer-readable instructions; the at least one processor executes the computer-readable instructions so that the computing device cluster executes the code generation method according to any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions; the computer-readable instructions are used to implement the code generation method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The method comprises computer-readable instructions; the computer-readable instructions are used to implement the code generation method according to any one of claims 1 to 10.

Images