Verification and provision of proactively generated code proposals

Abstract

Code completion proposals can be proactively obtained and verified. An event can be detected that triggers obtaining code completion proposals for inclusion in a code file being edited using an integrated development environment. Code completion proposals can be obtained. The characters of the code completion proposals can be compared with the characters added to the code file after the detection of the event that triggered the obtaining of the code completion proposals to determine whether the code completion proposals are valid. Then, valid code completion proposals can be displayed.

Description

Technical Field

[0001] Programming languages provide developers, designers, and other users with the ability to accurately specify the operation of various hardware or software designs for many different applications. Given the wide variety of programming languages available, these developers, designers, and other users may encounter or otherwise use code written in a programming language that the developer may not be very familiar with. Code development tools provide developers, designers, and other users with different capabilities to improve code performance and identify errors, which may help overcome the fact that developers are not familiar with the programming language (or the environment in which the programming language is deployed) in the above exemplary scenarios, so that high-performance code can still be written.

Brief Description of Drawings

[0002] [Figure 1] It is a logical block diagram showing code generation for code development according to some embodiments. [Figure 2] It is a logical block diagram showing a provider network that implements different services including a code development service according to some embodiments. [Figure 3] It is a logical block diagram showing code suggestion handling according to some embodiments. [Figure 4] It is a logical block diagram showing code suggestions according to some embodiments. [Figure 5] It is a logical block diagram showing an exemplary interface of a development environment according to some embodiments. [Figure 6] It is a logical block diagram showing an example of code suggestion development for subword regularization when training a machine learning model according to some embodiments. [Figure 7] It is a logical block diagram showing an example of code suggestion development for evaluation data set conversion according to some embodiments. [Figure 8] This is a high-level flowchart illustrating techniques and methods for implementing code proposal verification and proactive delivery in several embodiments. [Figure 9] This is an exemplary time for detecting, verifying, and displaying code completion suggestions, according to several embodiments. [Figure 10] This is a high-level flowchart illustrating techniques and methods for implementing the generation of code suggestions for input programming code, according to several embodiments. [Figure 11] This is a high-level flowchart illustrating techniques and methods for implementing constrained prefix matching to generate the next token prediction, according to several embodiments. [Figure 12] This is a logical block diagram illustrating different iterations of constrained prefix matching in several embodiments. [Figure 13] This is a high-level flowchart illustrating techniques and methods for implementing random token segmentation to train the following token prediction model, according to several embodiments. [Figure 14] This is a logical block diagram illustrating possible random tokenization of text by several embodiments. [Figure 15] This is a high-level flowchart illustrating techniques and methods for implementing the programmatic generation of evaluation datasets for a code generation system, according to several embodiments. [Figure 16A] The following are exemplary items of the evaluation datasets before and after transformation, according to several embodiments. [Figure 16B] The following are exemplary items of the evaluation datasets before and after transformation, according to several embodiments. [Figure 17] This block diagram shows an exemplary computing system in several embodiments.

[0003] Embodiments are described herein by example of multiple embodiments and exemplary drawings, but those skilled in the art will recognize that embodiments are not limited to the embodiments or drawings described. It should be understood that the drawings and their detailed description are not intended to limit embodiments to any particular form disclosed, but rather to encompass all modifications, equivalents, and substitutions that fall within the spirit and scope defined by the appended claims. Headings used herein are for constituent purposes only and are not intended to limit the scope of the description or claims. As used throughout this application, the word “may” is used in an allowable sense (i.e., meaning it is possible) rather than an obligatory sense (i.e., meaning it must be done). Similarly, the words “include,” “including,” and “includes” mean to include, but are not limited to, including. [Modes for carrying out the invention]

[0004] Various techniques for code generation and verification for code development are described herein. Sophisticated code development tools may rely on machine learning-powered features to assist in the design and development of new applications, systems, or services. Machine learning is used to truly improve the user experience with these features. Utilizing The speed and quality of features can depend on various aspects of their development and implementation. One such machine learning-powered feature for code development is code suggestions, which can generate code and recommend it to developers. Techniques to improve the speed and quality of machine learning-powered code suggestions can improve the user experience and the quality of applications, systems, or services generated using the features.

[0005] Figure 1 is a logical block diagram illustrating code generation for code development in several embodiments. An integrated development environment 110, such as a development application implemented locally on a user device (e.g., a computer or laptop) or hosted as part of a provider network service, can utilize code suggestion generation 120 when a code file input is received. As will be discussed in detail below, code suggestion handling 112 can proactively acquire and verify code suggestions 116 before providing them to the display 104, as will be discussed in detail below with respect to Figures 3, 4, 8, and 9 (114). In this way, higher-latency programming language suggestion models 122 implemented as part of code suggestion generation 120 that provide better and usable code suggestions 116 can be employed, even if their latencies are longer than those of lower-latency but less precise models, because proactive requests can bring the apparent latency of code suggestions to 0 or near 0, while still ensuring through verification that the suggestions are still valid before the display 104 (in light of potentially changing contexts such as other code file inputs 102). As will be discussed below with respect to Figure 3, other techniques such as storing and providing paginated results can also reduce the latency required to wait for code suggestions to display 104.

[0006] Other performance improvements to the use and implementation of code suggestion generation 120 can enhance the quality and accuracy of suggestions without affecting model performance. For example, as will be discussed in detail below with respect to Figures 4, 11, and 12, constrained prefix matching techniques can improve suggestions in partial word scenarios without increasing the time required to generate suggestions in a meaningful way.

[0007] Code proposal development 130, such as training 134 and deployment 136 of the programming language proposal model 122, can improve the impact on code proposal performance. For example, techniques for modifying the training dataset to train with subwords in mind, as will be discussed in detail below with respect to Figures 6, 13, and 14, can improve the code proposals made in those scenarios. Furthermore, code proposal generation 120 and evaluation of model 122 can be improved by increasing the number of high-quality evaluation datasets 132 that can be programmatically generated from other evaluation datasets, as will be discussed in detail below with respect to Figures 7 and 15-16B.

[0008] It should be noted that the above description is not intended to be limiting, but is merely provided as an example of an integrated development environment, code suggestion generation system, and tools for code suggestion development. Various other embodiments may also implement these technologies, as will be discussed in detail below.

[0009] This specification then includes a general description of a provider network that may implement code development services. Next, various examples of code development services are considered, including different components / modules, or configurations of components / modules, that may be adopted as part of implementing code development services in a provider network. Then, several different methods and techniques are considered, some of which are shown in the attached flowcharts. Finally, a description of an exemplary computing system in which various components, modules, systems, devices, and / or nodes may be implemented is provided. Various examples are provided throughout this specification.

[0010] Figure 2 is a logical block diagram showing a provider network implementing different services, including code development services, in several embodiments. The provider network 200 (which in some implementations may be referred to as the “cloud provider network” or simply “cloud”) refers to a pool of network-accessible computing resources (such as compute, storage, and networking resources, applications, and services), which may be virtualized or bare metal. The provider network 200 can provide convenient on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adapt to variable loads.

[0011] The provider network 200 can be formed as several regions, where a region is a distinct geographical area where the cloud provider clusters data centers. Each region may contain two or more availability zones connected to each other via a private high-speed network, such as fiber optic connectivity. An availability zone (also known as an availability domain, or simply a “zone”) refers to an isolated failure domain, comprising one or more data center facilities having separate power, separate networking, and separate cooling from those in other availability zones. Preferably, availability zones within a region are located far enough apart from each other that the same natural disaster should not bring two or more availability zones offline simultaneously. Customers can connect to the availability zones of the provider network 200 via publicly accessible networks (e.g., the internet, cellular networks). Regions are connected to a global network that includes a private networking infrastructure (e.g., fiber optic connectivity controlled by the cloud provider) connecting each region to at least one other region. The provider network 200 may deliver content from points of presence outside these regions but networked with them, via edge locations and regional edge cache servers. This compartmentalization and geographical distribution of computing hardware enables Provider Network 200 to provide customers with low-latency resource access on a global scale with a high degree of fault tolerance and stability.

[0012] As described above, Provider Network 2 00 may implement various computing resources or services such as code development services 210, and other services 230 which may be any other type of network-based services including various other types of storage (e.g., database services or object storage services), computation, data processing, machine learning, analysis, communication, event handling, visualization, and security services (not shown).

[0013] In various embodiments, the components shown in Figure 2 may be implemented directly within computer hardware (e.g., a microprocessor or computer system) as instructions that can be executed directly or indirectly by the computer hardware, or using a combination of these techniques. For example, the components of Figure 2 may be implemented by a system comprising several computing nodes (or simply nodes), each of which may be similar to the embodiment of the computer system shown in Figure 17 and described below. In various embodiments, a function of a given system or service component (e.g., a component of the code development service 210) may be implemented by a specific node or distributed across multiple nodes. In some embodiments, a given node may implement functions of two or more service system components (e.g., two or more data store components).

[0014] The code development service 210 may be implemented by a provider network 200 in some embodiments. The code development service 210 may implement various features for writing code for different systems, applications, or devices, and may provide features for recommending, identifying, reviewing, building, and deploying code. For example, the code development service 210 may implement a development environment 211. The code development environment 211 may provide various code input tools (e.g., text, diagram / graphics-based application development) for specifying, calling, or otherwise writing (or having written) code for different hardware or software applications.

[0015] The code development service 210 may implement code suggestion delivery 214 that may implement various computing resources for hosting and implementing code suggestions 213 in a scalable manner to deliver on-demand code suggestions across a large number of clients using a high-performance machine learning model for high-quality code suggestion results. For example, code suggestion delivery 214 may implement workload distribution and request management features for timely handling and returning code suggestions to provide real-time code suggestions with little or no apparent latency to code suggestion handling 220 (whether inside or outside the provider network 200).

[0016] To avoid forcing the development environment to wait for multiple code suggestions to be transmitted in a single communication, in some embodiments, code suggestion delivery 214 implements a pagination feature for code suggestions, allowing multiple code suggestions to be delivered over time via multiple communications from a host or other computing resource implementing code suggestions 213 to recipient development environments 219 and 211. In this way, valid code suggestions can be generated, presented, and then updated as more suggestions are received. Such techniques provide a simulated streaming experience without actually requiring bidirectional streaming supported by the development environment. In this way, the benefits of fast delivery and updating of code suggestions can be provided without introducing additional requirements to the development environment, which may not necessarily be maintained by the operator of the provider network 200.

[0017] To implement pagination, code suggestions may be stored in the service 210 when generated, and then returned via multiple interactions by leveraging a pagination token accompanying a request for code suggestions to allow additional code suggestions to be retrieved from storage and sent back to the development environment 219 or 211.

[0018] In various embodiments, the code suggestion 213 may generate code suggestions based on text input in the development environment 211 or 219, as will be discussed in detail below with respect to Figure 3 (for example, when code is entered into the development environment 211 or 219, it may utilize a plugin or other connection that can provide real-time analysis and code suggestions). The code suggestion 213 may use a generative model, a machine learning model such as a Generative Pre-trained Transformer (GPT), that has been trained to generate code suggestions. The generative model is often trained on a large corpus of data for a particular task. When generating code recommendations, this corpus (for example, from the code suggestion code repository 215 or other code repositories used to train the generative model) may consist of code repositories or snippets from various sources. Depending on the source or owner, the code may be subject to certain licenses that require attribution to any use or reproduction. Since the generative model can sometimes reproduce verbatim or near-verbatim matches to the training data, metadata to attribution to the original source may also need to be provided as part of the suggestion. Code suggestion metadata (not shown) may provide the ability to provide metadata for the code suggestions that may be provided.

[0019] The code development service 210 may implement (or have access to) a code repository 215. The code repository 215 may store various code files, objects, or other code that can be interacted with by various other features of the code development service 210 (e.g., the development environment 211 for writing, building, compiling, and / or testing code). In some embodiments, the code repository 215 may implement various versioning and / or other access controls to track and / or maintain consistent versions of collections of code for various development projects. In some embodiments, the code repository may be stored or implemented outside the provider network 200 (e.g., hosted on a private network or other location).

[0020] The code development service 210 may implement interfaces for accessing and / or utilizing various features of the code development service 210. Such interfaces may include various types of interfaces, such as command-line interfaces, graphical user interfaces, and / or program interfaces (e.g., application programming interfaces (APIs)), to perform requested operations, including the operation of the development environment 211. An API refers to an interface and / or communication protocol between a client and a server, where if a client makes a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In the context of a cloud provider network, an API provides a gateway for customers to access the cloud infrastructure by enabling them to retrieve data from the cloud provider network or trigger actions within the cloud provider network, and enables the development of applications that interact with resources and services hosted within the cloud provider network. APIs can also enable different services within the cloud provider network to exchange data with each other.

[0021] Generally speaking, client 250 may encompass any type of client that can be configured to submit network-based requests, including service requests (e.g., requests for code lookups or suggestions), to provider network 200 via network 260. For example, a given client 250 may include a preferred version of a web browser, or a plug-in module or other type of code module that can run as an extension to or within the execution environment provided by the web browser. Alternatively, client 250 may encompass any other application that can utilize resources within provider network 200 to implement an application (or its user interface), a media application, an office application, or various other applications. In some embodiments, such an application may include sufficient protocol support (e.g., for a preferred version of the Hypertext Transfer Protocol (HTTP)) for generating and processing network-based service requests without necessarily implementing full browser support for all types of network-based data. That is, client 250 may be an application that can directly interact with provider network 200. In some embodiments, the client 250 may generate a network-based service request according to a Representation State Transfer (REST) ​​style network-based service architecture, a document or message-based network-based service architecture, or another preferred network-based service architecture.

[0022] In some embodiments, client 250 may provide other applications with access to the provider network 200 in a manner transparent to those applications. For example, client 250 may integrate with a code development service 210. However, the operating system or file system may present applications with different storage interfaces, such as the traditional file system hierarchy of files, directories, and / or folders. In such embodiments, applications may not need to be modified to utilize the storage system service model. Instead, the details of the interface to the data storage service may be coordinated by client 250 and the operating system or file system, rather than by applications running within the operating system environment.

[0023] Client 250 can transmit network-based service requests to and receive responses from provider network 200 via network 260. In various embodiments, network 260 may encompass any preferred combination of networking hardware and protocols necessary to establish network-based communication between client 250 and provider network 200. For example, network 260 may generally encompass various telecommunications networks and service providers that collectively implement the Internet. Network 260 may also include private networks such as local area networks (LANs) or wide area networks (WANs), as well as public or private wireless networks. For example, both a given client 250 and provider network 200 may be provisioned within enterprises that have their own internal networks. In such embodiments, network 260 may include hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall / security software, etc.) necessary to establish networking links between a given client 250 and the Internet, and between the Internet and provider network 200. It should be noted that in some embodiments, client 250 may communicate with provider network 200 using a private network rather than the public internet.

[0024] In some embodiments, the provider network 200 may include hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall / security software, etc.) necessary to establish networking links between different components of the provider network 200, such as virtualization hosts, control plane components, and external networks 260 (e.g., the Internet). In some embodiments, the provider network 200 may employ Internet Protocol (IP) tunneling technology to provide an overlay network through which encapsulated packets can traverse the internal network using tunnels. IP tunneling technology may provide a mapping and encapsulation system for creating the overlay network and may provide separate namespaces for the overlay layer and the internal network layer. Packets in the overlay layer may be checked against a mapping directory to determine what their tunnel target should be. IP tunneling technology provides a virtual network topology, and the interface presented to client 250 can be attached to the overlay network so that the IP address runs within the virtual space, by communicating with a mapping service that knows where the IP overlay address is located, once client 250 provides the IP address to which it wants to send packets.

[0025] The perceived latency of code suggestions can reduce their use as a feature. For example, if a user has to wait a detectable period after requesting a code suggestion, the user workflow may be interrupted. To eliminate perceived latency, code suggestions can be prefetched. However, since the input context may have changed since the code suggestion was requested, techniques can be implemented to validate proactively retrieved code completion suggestions, ensuring that a given recommendation is no longer consistent with the current state of the code. Figure 3 is a logical block diagram illustrating code suggestion handling in several embodiments.

[0026] The code suggestion handling 220 may implement an automated suggestion event detection 310 that can detect events by evaluating keystrokes 342, elapsed time, special keys, or user-specific data. In various embodiments, this information may be maintained as part of user-specific state that can be updated or reset when a code suggestion request is submitted. For example, keystrokes, elapsed time, or other metrics may be reset. Special keys may also be trigger events (and may be evaluated in combination with other criteria such as elapsed time). For example, events that trigger the acquisition of code completion suggestions may include the input of "{", "[", "(", ":", "ENTER KEY", or "TAB KEY") and an elapsed time threshold. In some embodiments, the automated suggestion event detection 310 may use client-specific events, such as entering certain keys or characters in a client-specific pattern (or being configured / described by the client in a request that constitutes the suggestion handling 220).

[0027] The code suggestion request execution 320 may handle the formation, assembly, transmission, and processing of the response from the code suggestion 213, including obtaining a code completion suggestion and sending a request 322 to process the returned code suggestion 324. For example, the code suggestion request execution 320 may obtain a context window of tokens (e.g., N previous tokens before the cursor) from the file state 340, as shown in 344. In some embodiments, file and other context information may be transmitted, as provided by the file and other context extraction 350.

[0028] File and other context extraction 350 may employ different techniques to obtain file and other context information outside the context window (e.g., outside N previous tokens). For example, file context may be obtained from the same file from which the code suggestion is being generated for inclusion. Information that may be obtained for file context may include boundaries within the current scope (e.g., code and comments restricted by the current function to provide local context), class-level information including class declarations, class constructors (e.g., __init__ function), and function-level information for all other public or protected methods defined on the class, and function-level information including all functions declared in the current file on both sides of the cursor. In some embodiments, signatures, docstrings, and return statements may be extracted, and / or variable-level information including all previous variable declarations visible to the current generation focus may be extracted.

[0029] Other contexts that can be extracted in 350 may be project contexts. In modern code development, classes and functions are typically defined in hierarchically structured files. A simple backward context does not include information outside of the current file, which can lead to certain scenarios where machine learning models are less likely to generate correct code. Code files use imported classes / functions / variables vinegarru Often Therefore, adding this context can significantly improve code generation performance. Accordingly, in some embodiments, an in-project context may be added, in which case all imported classes, functions, and variables from the same project are used to obtain code completion suggestions.

[0030] Other contexts that may be extracted in 350 may be out-of-project contexts. Out-of-project contexts can refer to classes / functions / variables imported into the current file from other packages. This can have an impact on proposal quality when the imported package is under a zero-shot setup (e.g., when the pre-trained model has no prior knowledge of the package). Therefore, other contexts can be obtained by scanning for out-of-project contexts for packages not included in the pre-training data and including the corresponding classes / functions / variables as context in the request.

[0031] File and other context extraction 350 may perform regular expression-based searching (for example, for keywords such as "import") and extraction to obtain the various types of contexts considered above. In some embodiments, parsing-based extraction may be used (for example, by generating a symbol tree or other parsing graph of the code to obtain other context information).

[0032] The code proposal request execution 320 may interact with code proposals provided in a paginated format. For example, a response 324 to a request for code proposals may include a pagination token indicating that further proposals may be retrieved. The code proposal request execution 320 may further proceed to validate and provide proposals via code proposal validation 330, and also submit a subsequent request 322 with a pagination token to obtain further code proposal results, which may then be returned, validated, and provided. In this way, multiple proposals can be made, allowing for different execution times for the generation of code proposals, including potentially better code proposals that may be provided while the user is reviewing the initial proposals.

[0033] The file state 340 may provide information at various stages and, in some embodiments, may include both the code file and its associated metadata. The file state 340 may also provide information for code suggestion verification 330, such as the current character before the cursor.

[0034] The code proposal verifier 330 may verify the received code proposal before providing it for display. For example, the code proposal verifier may use one or more verification criteria to determine whether the added characters in the code proposal match or approximately match the characters prior to the cursor (and added after the time the code proposal request was made). A valid code proposal may be provided for display, as shown in 352. In some embodiments, as shown in 354, the acceptance (or rejection) of these proposals may be received and passed to or included in the file state 340, as shown in 356.

[0035] Coded proposal verification 330 may identify and display valid coded proposals (348) and handle the approval or rejection of proposals.

[0036] The code proposal 213 can be implemented to provide various code proposals in different scenarios. Figure 4 is a logical block diagram showing code proposals in several embodiments. The code proposal request 401 may be received by the tokenizer 410. For example, various different tokenizers or tokenization techniques may be used. Tokens may be individual words in a sentence, including the preceding space character "[space]word" as a token. Punctuation marks, whitespace, carriage returns, or various other characters may also be individually grouped or considered as tokens.

[0037] The programming language token prediction model 420 may use the provided code 422, as well as other contexts 424 such as a token window or a file context outside of other files, which can be obtained using techniques such as regular expressions or parsing, as discussed above with respect to Figure 3. The token prediction model 420 may be specified for a programming language (e.g., Model A for language A, Model B for language B, etc.). Each of them may be trained with code in its respective programming language and other contexts to generate recommendations. The programming language token prediction model 420 may also be trained with other contextual information (e.g., the same file, the same project, or other contexts), as discussed above with respect to Figure 3.

[0038] In some embodiments, the code development service 210 may support a custom programming language model. For example, training data or code data from a user's specific code repository may be provided to train a custom programming language model, which may then be used for code suggestions.

[0039] The predictions may be provided to selection 430, and one may be selected based on its confidence score to be provided as a code suggestion. In some embodiments, multiple predictions may be provided in a paginated format or other multiple result format, as discussed above with respect to Figure 3.

[0040] One scenario in which machine learning models generating text recommendations may encounter is when the input contains partial words, such as "Syst". In these scenarios, machine learning models tend to provide poor predictions and therefore poor recommendations (e.g., producing gibelish or incoherent generation). This occurs because the model only sees word tokens as input units. To overcome this scenario, we constrain the generation to backtrack to the last complete token and match the prompt suffix, which is "Syst". As discussed below, constraining the generation helps improve accuracy for subword data metrics without compromising gains over general evaluation sets.

[0041] Given a string prompt, inconsistencies from normal decoding can be caused by the prompt's suffix, which may potentially occur with subwords that are not complete tokens. Matching an input string suffix can be performed using all available tokens that begin with the suffix or with all available tokens that begin with the suffix. In some embodiments, matching is performed efficiently using a character tridata structure (e.g., using a native Pytorch array as a node list for fast concatenation). Based on the list of matching tokens, other tokens may be masked during the next token prediction, thus ensuring that the generation matches the suffix character by character. Further latency optimizations can be achieved by caching very frequent suffixes, such as single spaces, by holding a Boolean mask. For each step matching a suffix, after each token generation step, the matching tokens are removed from the left of the suffix (by character), and constrained generation is performed until the suffix becomes an empty string. In some embodiments, subtokens (e.g., suffixes) are determined by using the same pre-token partitioning strategy as the tokenizer 410 that performs partitioning using word boundaries, which enables efficient backtracking for character matching because it can be deterministically known that no subtoken can cross a pre-token boundary.

[0042] The constrained prediction generator 440 may implement these techniques to provide code suggestions, as will be discussed in more detail below with respect to Figures 11 and 12. For example, the constrained prediction generator 440 may be a feature that is enabled (or disabled) for code suggestions. In some embodiments, the constrained prediction generator 440 may be configured (e.g., via a suggestion request) to utilize a specified maximum number of backtrack tokens (which may be dynamically determined for up to a maximum number of code suggestions). The constrained prediction generator 440 may receive tokenized input data from the tokenizer 410 and then determine backtrack tokens and partial tokens.

[0043] The constrained prediction generator 440 can then, in various embodiments, identify one or more possible tokens that match a subtoken that can be identified from the possible tokens. For example, a possible token could be a vocabulary of a language (e.g., programming or human) that may have different words that can be predicted. A match can be identified when the subtoken matches either the beginning or the end of a possible token (for example, a possible token matches when the subtoken [SYS] matches the beginning [SYS*, or the end *SYS] of the possible token). Matching can be performed using a tri structure as considered above and below. A cache for common matches can also be utilized.

[0044] The constrained prediction generator 440 then performs one or more iterations of the next token prediction, filtering for identified possible matches, evaluates the remaining next token prediction using the partial token, and may subtract matching characters from the partial token until all characters from the partial token match. The code proposal may then be provided (for example, to a prediction selector 430 which may simply forward or send the next token prediction as a code proposal 401).

[0045] Figure 5 is a logical block diagram showing exemplary interfaces of a development environment according to several embodiments. The integrated development environment interface 500 may be implemented on a client of the code development service 210, as depicted in Figure 2, or may be hosted as part of the code development service 210, as depicted in Figure 2. The integrated development environment interface 500 may implement a code editor 510 (e.g., a text editor) that can enable a user to input code in a programming language. The code suggestion feature 213 of the code development service 210 may parse the input characters to determine a code suggestion 520, which may be displayed and added, as shown in 522. Although not shown, various other information about the suggestion may be displayed, such as the source of the code, the licensing information of the code, and / or various other code metadata (e.g., style guidelines).

[0046] Code suggestion and other next word or token prediction techniques may encounter scenarios where a partial token is the closest contextual input for code suggestion (or other next word / token prediction). In the following example of Java code, code suggestion occurs after the cursor, and the cursor is " <t>Consider the following exemplary scenario, which can be expressed as "[...]." GetRecordsResult result = streamClient.getRecords(streamName); while(result.getNextRecordMarker()!=null){ / / (2) System.out.println(result.getNextRecordMarker()); result = streamClient.getRecords(streamName); S <t><- current cursor The expected, and most likely, thing to do after the cursor is System, a very common module referenced in Java, which also appears in the context above. However, since "S" is a partial token, the current cursor code proposal provided just before may not handle partial tokens well. Consider the following exemplary input with the proposed code underlined after the cursor. / / Input -> S S <t> sleeper sleeper = new Sleeper(1000); / / Input -> Sy Yes <t> strace.endSection(); / / Input -> Sys Sys <t> out.println(result.getNextRecordMarker()); / / Input -> Syst Syst <t> en.sleep(1000); / / Input -> Syste Syste <t> .sleep(1000);

[0047] In the example above, which creates longer subtokens, possible suggestions may still not provide the expected "System" result. This can happen because the machine learning model used to generate the suggestions was not trained on subtokens such as "S" or "Sy".

[0048] This lack of training can arise due to how the tokenizer breaks down the training dataset for the machine learning model used to generate the proposal. Consider the following example token generated from the input text (to the left of "->"). Name -> [' Name'] Sleeper -> [' S', 'le', 'eper'] / / Prefix S<> Systrace -> [' Sy', 'str', 'ace'] / / Prefix Sy<> Sysout -> [' Sys', 'out'] / / Prefix Sys<> Systen -> [' S', 'yst', 'en'] / / Prefix Syst Syste -> [' S', 'yst', 'e'] / / Prefix Syste

[0049] To address this problem, randomized segmentation of several tokens can be applied to training datasets for training or fine-tuning machine learning models for code (or other text) proposals. For example, one example of code text might be as follows: System.out.print("test"); If we tokenize this directly, it will look like this: [' System', '.', 'out', '.', 'pr', 'in', 'ln', '(”', 'test', '");'] Instead, random splits may be inserted into the sentence, dividing it into two parts. 'System.out.prinln("test");'-> ['S', 'system.out.prinln("test");'] Next, each segment is tokenized individually, and then they are concatenated, which provides the following tokenization result. [' S', ' y', 'st', 'em', '.', 'out', '.', 'pr', 'in', 'ln', '(”', 'test', '");'] This can be used as part of a training dataset for pre-training or continuous fine-tuning.

[0050] Implementing randomized segmentation for tokens can, in various embodiments, result in various configurations of subtokens with respect to the original token (for example, from "Sy" to "System"). 」 A machine learning model can be trained on how to construct (). Such techniques used to train the machine learning model can significantly improve the accuracy of partial token completion without compromising the accuracy of full token prediction scenarios. Furthermore, these techniques can be implemented without slowing down the machine learning model during the inference phase.

[0051] Figure 6 is a logical block diagram showing an example of code proposal development for subword regularization when training a machine learning model, according to several embodiments. The random subword tokenizer 610 and model training are tools, systems, or features of the code proposal development 217. as implementation So In other embodiments, a separate training system, application, or service, such as a machine learning service implemented as part of the provider network 200 in Figure 2, may implement these techniques, as well as the techniques discussed below with respect to Figures 13 and 14.

[0052] The random subword tokenizer 610 may acquire training data 602 and randomly segment it, as discussed below with respect to Figures 13 and 14 for model training 620. For example, a request may be received to generate partially tokenized training data, which may specify a storage location or other information describing a source training dataset containing text data. According to some embodiments, the random subword tokenizer may determine multiple tokens from text data. For example, tokens may be generated from input text by applying various different tokenizers. For example, tokens may be individual words in a sentence, including the preceding space character "[space]word" as a token. Punctuation, whitespace, carriage returns, or various other characters may also be individually grouped or considered as tokens.

[0053] In some embodiments, the random subword tokenizer 610 may randomly select different tokens from a plurality of tokens and segment them into their respective subtokens. For example, a subword regularization technique may be performed to sample or identify different tokens for non-optimal segmentation. Such a technique may include randomly selecting tokens from tokens determined for text data (according to a configurable variable or parameter which may be expressed as a percentage value, for example), where the percentage value indicates the probability that any one token will be selected for random segmentation. If a token is selected, the token may be randomly segmented into subtoken components which will be treated as tokens, instead of the source token from which the token is generated.

[0054] Model training 620 can implement various machine learning training frameworks that allow machine learning jobs, applications, or programs to be run on the initial model 604 using training datasets generated by the random subword tokenizer 610. Note that training can be performed to train a model from scratch or to train with given deregularization checkpoints. For example, the initial model may include various neural network-based machine learning models that are already pre-trained and therefore provided for fine-tuning, or it may be a new model that is not pre-trained. Various different hyperparameters or other configurations for model training may be specified as part of a training job or request and used to perform training on the initial model. Upon completion, model training 620 may provide a trained model with subword regularization, as shown in 606.

[0055] Another tool, system, or feature of Code Proposal Development 217 may be a Programming Language Transmitter 710 that can translate a source evaluation dataset of one programming language into another programming language. High-quality evaluation datasets are time-consuming to create and typically require a large amount of annotator time and effort. This is also true for execution-based function completion evaluation sets. In various embodiments, programmatic test translation tools from a source programming language such as Python to other target programming languages ​​are applicable to tests that perform exactness evaluations based on the return values ​​of functions with ground truth values ​​(hence, value-oriented translations). These embodiments can be used to translate many test cases, which helps reduce annotation time and increase the number of evaluation datasets in order to build and test additional code generation systems for many different languages. These techniques are widely applicable and can be used to translate existing datasets such as MBPP (Most Basic Python Programming) into Javascript, Java, Typescript, Ruby, Go, C#, or any other programming language for which a translation rule set is generated.

[0056] In some embodiments, the conversion process begins with inferring the type of function arguments, which can be done by examining the argument values ​​within each test case. This includes mapping types from Python to each language, such as from "list" to "Arraylist" in Java, or from "dictionary" to "Hash Map". It will be held The values ​​of different test cases can have different types, and therefore, a common superclass for all observed types of each argument can be inferred according to the type hierarchy. Since there can also be many levels of types (due to containers such as lists or sets), types can be recursively inferred to be consistent at each level. For example, "list of lists" and "list of objects" have the common type "list of objects". The type of the expected return value can also be inferred by examining the expected return value in the test case, which also matches the value of the function executed using the given inputs of that test case.

[0057] In addition to types, this involves converting arguments and return values ​​from a source programming language to a target programming language by generating strings that represent objects in the target language that can be parsed by their respective interpreters / compilers. For example, Python's [1,2] is converted to 'Arrays.aslist(1, 2)', or Python's {1 : 2, 3: ["foo", "bar"]} is converted to 'new HashMap(){{put(1, 2);put(3, Arrays.aslist("foo", "bar"))', which has recursive support for arbitrary nesting structures.

[0058] For test case transformation, in some embodiments, all information regarding the return type and arguments / expected return values ​​can be combined to construct code that represents the input / output objects in the appropriate target programming language using an appropriate comparator for equivalence.

[0059] In addition, in some embodiments, a source programming language prompt string is included, which includes a function signature and docstring, including input / output examples. teeth The docstring can be converted to a prompt string in another target programming language. The style of the function signature can be mimicked in each language, along with the appropriate return / argument types where applicable, and function / argument / class names can be converted to a stylistically appropriate format (camel case or Pascal case). The docstring can be formatted so that the input and output look as close as possible to the input / output format for that particular language.

[0060] Figure 7 shows several embodiments. Review This is a logical block diagram illustrating an example of code proposal development for value dataset conversion. As discussed above, and as will be discussed in detail below with respect to Figures 15-16B, the programming language converter 710 can convert a given evaluation dataset from a source programming language to a target programming language. For example, the programming language converter 710 may receive a conversion request 702. The conversion request 702 may specify the source programming language and target programming language of the evaluation dataset, as well as the storage location or other access information of the source evaluation dataset, such as the source evaluation dataset for programming language A772, and the target storage location, format, and / or other access information for generating a new evaluation dataset, such as the storage location of the new evaluation dataset for programming language B774.

[0061] The programming language translation 710, in 720, the test statement translation 730, and the natural language translation 750, may utilize different translation techniques for different parts of the items in the evaluation dataset, such as techniques for inferring or mapping the types of function signatures (e.g., recursively). Each of these features may utilize specific translation rules, mappings, and / or type hierarchies for a given source and target programming language.

[0062] Function signature conversion 720 can identify function signatures in source 772 by parsing items in an evaluation dataset according to a parser or rule set for the first (source) programming language and locating the function signatures. In the Python programming language, for example, a search for "def" (e.g., a regular expression search) may be performed to locate the function signature, and the function signature may be delimited by various other symbols (e.g., arguments may be enclosed in parentheses). Once the function signature is located, different techniques may be used to determine what type each argument or parameter of the function is. For example, test cases for the function may identify the values ​​of the arguments. To complete the conversion, one or more specific to the conversion of function signatures from the source programming language to the target programming language are used. multiple Mapping rules may apply.

[0063] Test statement transformation 730 can transform test statements using knowledge determined as part of transforming function signatures 720. For example, using the argument format of function signatures in the source, various test values ​​can be extracted from source test statements and inserted into the target programming language version of the test, which may be obtained as a template test statement that accepts the arguments and triggers an error or other instruction if the test statement fails.

[0064] Natural language translation 750 may be implemented as part of translating prompts from source 772 into a target programming language evaluation dataset 774. For example, prompt translation may include changing features such as symbols used to indicate code comments (e.g., non-executable statements in code), e.g., changing from """ to " / * *". Prompt translation may also include modifying natural language statements to replace source programming language terms with target programming language terms such as "Write a function in Python" to "Write a function in Java", or changing between terms such as "none" to "null". In some scenarios, translation rules may remove unnecessary or uncovered table source programming language specific statements.

[0065] The function body in the target programming language may be generated by sending a request from 740 to code suggestion 213, which may receive request 704 and return the generated code 706. In some embodiments, the request may include a converted prompt and a converted function signature for a test item.

[0066] Examples of verifying and proactively providing the code proposals discussed above with respect to Figures 2-7 are given in relation to an example of a code development service. Various other types of code development tools, systems, or applications may implement these techniques. Figure 8 is a high-level flowchart showing techniques and methods for implementing the verification and proactive provision of code proposals in several embodiments. These techniques, as well as those discussed below with respect to Figures 9-16B, may be implemented using various components of a provider network, as discussed above with respect to Figures 2-7, or other types or systems that implement code development tools or other applications.

[0067] As shown in 810, in some embodiments, events can be detected that trigger the acquisition of code completion suggestions for inclusion in a code file being edited using an integrated development environment. In some embodiments, the events that trigger the acquisition of code completion suggestions may be based on one or more criteria. For example, a keystroke count (since the last code completion suggestion request was made) may be maintained. Only this keystroke count may trigger an event when the number of keystrokes exceeds a threshold. In some embodiments, other criteria may be considered. For example, the time elapsed since the last trigger may also be used, which may allow code suggestions to be acquired after a certain period has elapsed since the last suggestion was made. In some embodiments, a combination of criteria (e.g., keystrokes and elapsed time) may be used. In some embodiments, event triggers may be user or client specific, based on heuristics such as typing or using a particular character or key (e.g., after selecting the TAB key for indentation).

[0068] As shown in 820, in some embodiments, the generation of code completion suggestions may be triggered. In some embodiments, the code completion suggestion may be based on the character immediately preceding the cursor at a first time when an event triggering a request for a code completion suggestion is detected, and the code completion suggestion includes, at the first time, the suggested character to be entered into the code file immediately following the cursor. In some embodiments, the code suggestion may be implemented and executed locally (e.g., by a local subsystem). In some embodiments, the code suggestion may be generated remotely (e.g., as a feature of the code development service 210 in Figure 2).

[0069] As shown in 830, a comparison between the number of characters proposed and the number of characters actually entered into the code file after a first time step may be used to determine whether one or more validation criteria are met. For example, the validation criterion could be an exact match, as discussed below with respect to Figure 9. In other embodiments, the validation criterion could allow for a fuzzy match or a near match (e.g., a 3 out of 4 character match). If no match is found, the code proposal may be discarded, as shown in 850. Otherwise, the code proposal may be displayed, as shown in 840.

[0070] In some embodiments, code completion suggestions can be accepted or rejected by the user, which in turn can trigger recommendations for further code completion suggestions.

[0071] Figure 9 shows exemplary timelines for detecting, verifying, and displaying code completion suggestions according to several embodiments. At time T1, the input code 910 in the editor is shown along with the cursor's position. An event is detected that triggers the acquisition of a code suggestion. At time T2, the input code 920 has been modified with additional characters added, as indicated by the moved cursor. The code suggestion 930 may be verified at T2 by comparing a certain number (e.g., 4) characters with the additional character "temp".

[0072] Code suggestions for programming languages ​​may, in various embodiments, be provided by the code suggestion feature of a service such as a code development service 210, or as a standalone code generation system. Many of the techniques considered above and below for improving the performance of the various stages of code suggestion can be integrated with code suggestions generated using techniques such as those shown in Figure 10. Figure 10 is a high-level flowchart showing techniques and methods for implementing the generation of code suggestions for input programming code, according to several embodiments.

[0073] As shown in 1010, in some embodiments, a request may be received to generate code suggestions for input programming code. For example, in some embodiments, a code suggestion request may be generated as part of an eager or expected code suggestion request technique, as discussed below with respect to Figures 3, 8, and 9, or as a manual request for recommendation. In some embodiments, the request may be used to perform various techniques, such as generating code suggestions to provide a transformed function body for evaluation dataset transformation, as discussed below with respect to Figures 15–16B.

[0074] As shown in 1020, in some embodiments, tokens may be determined from input programming code. Various tokenization techniques discussed above and below may be implemented. For example, various different tokenizers or tokenization techniques may be used. Tokens may be individual words in a sentence, including the preceding space character "[space]word" as a token. Punctuation marks, whitespace, carriage returns, or various other characters may also be individually grouped or considered as tokens.

[0075] As shown in 1030, in some embodiments, the next token prediction for the input programming code can be generated by applying a machine learning model, trained to generate the next token prediction for a programming language corresponding to the programming language of the input programming code, to the tokens of the input programming code. This machine learning model may be trained using randomized token segmentation, as discussed above with respect to Figure 6 and below with respect to Figures 13 and 14. Various different types of machine learning models can be used, such as generative pre-trained transformers (GPTs), sequence-to-sequence models, or other neural network-based models such as long short-term memory (LSTMs).

[0076] As shown in 1040, in some embodiments, one of the following token predictions may be selected according to its respective confidence score to be returned as a code suggestion for the input programming code. In some embodiments, multiple recommendations may be generated, and several recommendations may be provided, including multiple recommendations (e.g., the top three according to the confidence store).

[0077] As discussed above with respect to Figures 1, 5, and 6, partial token scenarios can cause machine learning models to generate incorrect text proposals. Several techniques can be applied to improve the training-side problem, as discussed above with respect to Figure 6 and below with respect to Figures 13 and 14, but inference time techniques can also be implemented. In various embodiments, constrained prefix matching for generating the next token prediction can be used in such situations to correct partial token proposal errors. Figure 11 is a high-level flowchart showing techniques and methods for implementing constrained prefix matching for generating the next token prediction by several embodiments.

[0078] As shown in 1110, in some embodiments, input text may be received for performing the next token prediction on the input text. For example, the input text may be received as part of a request for code suggestions, as considered above with respect to Figures 4, 8, and 10, and may be an amount of code written in a programming language in front of the cursor when the code suggestion request is triggered. The input text may also be received for other text prediction or completion scenarios (e.g., text autocomplete for various use cases such as text suggestions for drafting various documents, or providing input for other requests or forms in natural language).

[0079] As shown in 1120, word boundaries can be determined for the tokenizer of the input text. The rightmost boundary potentially contains a subtoken. This subtoken can be used as a prompt suffix to constrain the next token generation. Note: These word boundaries are larger units than tokens and are often referred to as pretokens. The last token may be a subtoken. For example, various different tokenizers or tokenization techniques may be used. A token can be an individual word in a sentence, containing the preceding space character "[space]word" as a token. Punctuation, whitespace, carriage returns, or various other characters can also be individually grouped or considered as tokens.

[0080] In some embodiments, a “pretoken” occurring immediately before a partial token may be identified. From these pretokens, the backtrack tokens may be determined. For example, starting with the pretoken immediately preceding the prompt suffix, one or more pretokens may be added to the backtrack tokens, which operate in reverse order of the token sequence, until the maximum number of backtrack tokens is reached or a special character (e.g., a carriage return) is reached. In some embodiments, the maximum number of backtrack tokens may be a configurable parameter for the next token prediction (e.g., as part of the request for the next token prediction or as a separate configuration request).

[0081] As shown in 1130, in various embodiments, one or more possible tokens matching a prompt suffix can be identified from the possible tokens. For example, the possible tokens may be a vocabulary of a language (e.g., programming or human) that may have different words that can be predicted. A match can be identified when the prompt suffix matches either the beginning or the end of the possible token (for example, a possible token matches when the prompt suffix [SYS] matches the beginning of the possible token [SYS*, or the end of the possible token *SYS]).

[0082] In some embodiments, one or more different data structures may be used to identify possible matching tokens. For example, a tri data structure may be used to store different possible tokens. A tri may be a search tree for prefixes (or suffixes), which may be string-indexed against a vocabulary of words, with each individual node containing a link to a suffix child node in each child node that adds an additional character to the suffix. Another example of a data structure that may be used to efficiently identify matching tokens may be a cache of possible matching tokens (e.g., as a boolean mask).

[0083] As shown in 1140, in some embodiments, the next token prediction may be filtered according to one or more identified possible tokens, and the next token prediction is generated by applying a machine learning model to the remainder of the input text, excluding the number of backtrack tokens corresponding to pre-tokens. For example, a given input to the machine learning model may have several input tokens (including subtokens) from the text before the cursor, such as 15 tokens where token 15 is a subtoken. If the number of backtrack tokens is 3, tokens 14, 13, and 12 (adjacent to token 15) may not be used as input for the next token prediction machine learning model, and as a result, the input may instead be the tokens preceding tokens 11 through 1 (and possibly including three more preceding tokens to compensate for the backtrack tokens) and the subtoken 15.

[0084] Given an input token, the result of the next token prediction may include several different token predictions with fluctuating confidence values. Token predictions that are not one of the identified possible tokens may be excluded from consideration. The highest remaining confidence score among the predictions may be identified.

[0085] For this remaining next token prediction, the number of characters matching the partial token may be subtracted from the left of the partial token, as shown in 1150. If no further characters remain, as shown in 1160, the next token prediction may be provided as the next token prediction, as shown in 1170. If none remain, another iteration of the next token prediction may be performed using the remaining characters until no more characters remain, as shown by the affirmative exit from 1160.

[0086] Figure 12 is a logical block diagram showing different iterations of constrained prefix matching according to several embodiments, as discussed above with respect to Figure 11. Iteration 1201 shows an input that includes the rest but does not include the backtrack token 1202 and the subtotal token (e.g., "[SPACE]SYSTE"). The filtered prediction with the highest confidence may be "[SPACE]SY". This can be used to subtract the matching character from the subtotal, as shown in iteration 1203, so that "STE" remains. In this iteration, the prediction may be "[SPACE]SYST". After subtraction, the subtotal is "E". In iteration 1205, the prediction may be "[SPACE]SYSTEM", which matches the remaining character "E", and the iteration ends, with the prediction "[SPACE]SYSTEM" being used as the next token prediction.

[0087] As discussed above with respect to Figures 1 and 6, training data for code proposals and other text generation systems can encounter difficulties when partial tokens are included as part of the input for text generation inference, which can occur in various autocomplete scenarios where text (e.g., code) is generated to automatically complete the next part of the text, given the input context of previously entered text up to several partial tokens (e.g., partial or incomplete words). To improve the ability of machine learning models to consider these scenarios, the training dataset may need to include data items that train these partial token scenarios. Figure 13 is a high-level flowchart illustrating techniques and methods for implementing random token segmentation to train the next token prediction model, according to several embodiments.

[0088] As shown in 1310, in some embodiments, text data may be received for training a machine learning model to predict the next text token given an input text token. For example, a request may be received to generate partially token-optimized training data, which may specify a storage location or other information describing a source training dataset containing text data. In some embodiments, this request may be received or specified as part of a training job submitted to a machine learning system or service, such as a machine learning service implemented as part of a provider network, like provider network 200 in Figure 2.

[0089] As shown in 1320, according to some embodiments, multiple tokens can be determined from text data. For example, tokens can be generated from input text by applying various different tokenizers. For example, tokens may be individual words in a sentence, with the preceding space character "[space]word" as the token. Punctuation marks, whitespace, carriage returns, or various other characters may also be individually grouped or considered as tokens.

[0090] As shown in 1330, in some embodiments, different tokens among a plurality of tokens may be randomly segmented into their respective subtokens. For example, subword regularization techniques may be performed to sample or identify different tokens for non-optimal segmentation. Figure 14, considered below, provides an example of random token segmentation. Such a technique may involve randomly selecting tokens from tokens determined for text data (according to a configurable variable or parameter which may be shown, for example, as a percentage value), where the percentage value indicates the probability that any one token will be selected for random segmentation. If a token is selected, the token may be randomly segmented into subtoken components which will be treated as tokens, instead of the source token from which the token is generated.

[0091] As shown in 1340, in some embodiments, a machine learning model may be trained to predict the next token given an input text token, using multiple tokens, each containing a subtoken, as a training dataset. In some embodiments, the trained machine learning model may be stored in a location specified in the training request. In some embodiments, the machine learning model may be deployed for different applications, including code suggestion or autocomplete applications for text, as considered above. Various different training techniques and machine learning model types may be used for next token prediction, such as sequence-to-sequence models or other neural network-based models like Long Short-Term Memory (LTSM).

[0092] Figure 14 is a logical block diagram illustrating possible random tokenization of text by several embodiments. Text data such as "New York" as shown in 1410 may be an example of a longer text string that can be tokenized for a training dataset, as considered above with respect to Figure 13. Each word, "New" and "York," may be a separate token. Since tokenization can be performed randomly, for example by using subword regularization techniques, different examples can be generated, as shown in 1420, 1430, and 1440 (each block being a subtoken). In these examples, a random selection is made as to whether tokens are selected for further segmentation (e.g., both in 1420, "York" in 1430, and "New" in 1440). Random segmentation may result in two or more segments, which in some embodiments may involve treating individual characters as subtokens.

[0093] As discussed above with respect to Figures 1 and 7, high-quality datasets for training and evaluating different systems, services, or applications, such as those implemented to provide code or other text suggestions as discussed above, can be difficult to obtain. Some datasets, such as evaluation datasets, can be highly specialized. For example, an evaluation dataset for a code suggestion system may depend on various code prompts or problems to be solved. A code prompt may describe a programming problem that can be taken up by the code suggestion system, and then have a corresponding part of the code generated to solve or satisfy that problem. For example, a prompt may be desirable to be a function that checks whether a given number is odd or even. It may include various exemplary assertions or unit tests that can be used to determine whether the generated code returns the correct values ​​that satisfy the tests. To increase the number of available high-quality evaluation datasets, techniques for generating new high-quality evaluation datasets may be very desirable.

[0094] Figure 15 is a high-level flowchart illustrating techniques and methods for implementing the programmatic generation of evaluation datasets for a code generation system in several embodiments. This technique can be implemented by various types of systems for testing, developing, or implementing code proposals or other code generation systems. In some embodiments, a programming language translation system may include a suite of tools, including tools for programmatically generating evaluation datasets for a code generation system.

[0095] As shown in 1510, in some embodiments, an evaluation dataset specified in a first programming language may be received, and different items in the evaluation dataset correspond to different evaluation tests of the code generation system. For example, one or more files, objects, locations, or other information for accessing and retrieving the evaluation dataset may be provided as part of a request to perform a conversion of the evaluation dataset from a first (e.g., source) programming language to a second (e.g., target) programming language. In some embodiments, multiple target programming languages ​​may be specified as part of the request, and thus multiple executions of the technology may be specified, as considered below.

[0096] As shown in 1520, in some embodiments, individual items of different items in a dataset may be translated into a second programming language. For example, prompt translation may include changing features such as symbols used to indicate code comments (e.g., non-executable statements in code), e.g., changing from """ to " / * *". Prompt translation may also include modifying natural language statements to replace source programming language terms with target programming language terms such as "Write a function in Python" to "Write a function in Java", or changing between terms such as "none" to "null". In some scenarios, translation rules may remove unnecessary or uncovered table source programming language-specific statements.

[0097] As shown in 1530, the function signatures of items in a first programming language can be translated into a second programming language. For example, a function signature can be identified by parsing items in an evaluation dataset according to a parser or rule set for the first (source) programming language and locating the function signature. In the Python programming language, for example, a search for "def" (e.g., a regular expression search) may be performed to locate the function signature, and the function signature may be delimited by various other symbols (e.g., arguments may be enclosed in parentheses).

[0098] Once a function signature is localized, different techniques may be used to determine the type of each argument or parameter of the function. For example, a test case of the function may identify the values ​​of the arguments. In Figure 16A, for example, the function signature 1612 may provide the dataset source item 1610 being transformed. The function signature 1612 may be localized (as considered above) and then evaluated to determine the values ​​of the arguments "cost, m, n". One of the test cases shows that the candidate function has an input such as "[[1, 2, 3], [4, 8, 2], [1, 5, 3]], 2, 2", which can be decomposed into cost as "[[1, 2, 3], [4, 8, 2], [1, 5, 3]]", m as "2", and n as "2". Thus, cost can be a list and m and n can be integers. The return value can also be inferred from the test, for example, "[[1,2,3],[4,8,2],[1,5,3]],2,2==8", where "8" is the desired return value for that test case. In some embodiments, return value determination may involve running both the source test item function and the test to identify the returned value.

[0099] To complete the conversion, one or more specific to the conversion of function signatures from the source programming language to the target programming language multiple Mapping rules may be applied. In the illustrated examples in Figures 16A and 16B, a mapping rule is applied in 1624 to convert the text of the Python function signature 1612 to Java, where "def" is "class MinCost{public static in MinCost(List <List <integer>>cost, int m, int n } It is converted as . As considered above, the determination of the argument can allow mapping of the determined source value type to the target value type explicitly declared in Java Example 1624 (e.g., "List", "integer", and "integer"). In some scenarios, the argument is a list of heterogeneous data (it could be a list as an element, a set, etc.). To perform the conversion, the inference technique can examine the data type by recursively looking at the elements. In such a technique, the most common type can be selected given all values. For example, if one of the argument values ​​is List <integer>The other argument value is a List. <double>If it is a type, the type is List <double>It should be. In another example, if one of the argument types is List <List <integer>> and the other argument type is List <HashMap<Integer,String> >If this is the case, the type is List <object>This is the result.

[0100] As shown in 1540, in some embodiments, test statements of items in a first programming language can be translated into a second programming language. Some knowledge determined as part of translating function signatures can be used to translate test statements. For example, various test values ​​can be extracted from 1618, such as translating "assert candidate" from "assert candidate" to class Main { public static void main(String[] args) throws Exception { if (!(MinCost.MinCost(Arrays.asList(Arrays.asList(1, 2, 3), Arrays.asList(4,8,2),Arrays.asList(1,5,3)),2,2)==8) throw new java.lang.Exception("Exception -- test case 0 did not pass");}" using the argument format "(cost, m, n)", and then inserted into the target programming language version of the test, as shown in test statement 1628. This can be repeated for each test.

[0101] As shown in 1550, in some embodiments, the body of the translated function signature can be generated in a second programming language according to prompts within an item used as input to a machine learning model trained to generate code in a second programming language. For example, as will be considered in detail below with respect to Figures 1, 2, 4, and 10, a code generation technique can take a given portion of the code to be generated or a natural language statement describing the code as input and generate the proposed code. Thus, a code translation system (or any other system, service, or application performing the techniques enumerated in Figure 15) can invoke the code generation system or a locally implemented machine learning model via an interface to generate function body code in a target programming language (for example, by specifying the target programming language to trigger the use of a machine learning model specific to that target programming language).

[0102] In various embodiments, the assembly of different converted item parts may be completed according to one or more conversion rules for items in the evaluation dataset in the source programming to the target programming language. For example, the ordering of parts may vary from one programming language to another. In Figure 16A, for example, the order of the test item is a function signature 1612, then a prompt 1614, then a function body 1616, and finally a test statement 1618, and the converted version of 1620 is a prompt 1622, then a function signature 1624, then a function body 1626, and finally a test statement 1628. Different programming languages ​​may have different required orderings that may be considered when assembling the converted version.

[0103] As shown in 1560, in some embodiments, individual converted items from different evaluation datasets within an evaluation dataset may be stored as part of a new evaluation dataset. For example, each item within an evaluation dataset may be a different file, document, or other object. When each new converted item is created, the corresponding different file, document, or object may be added to the target storage location for the new evaluation dataset. In some embodiments, various errors may trigger the storage of the source item in a separate storage location for notification and / or manual conversion (e.g., sending a notification that the source item should be reviewed).

[0104] The methods described herein may be implemented in various embodiments by any combination of hardware and software. For example, in one embodiment, the method may be implemented by a computer system (e.g., a computer system like the one in Figure 17) that includes one or more processors that execute program instructions stored on a computer-readable storage medium coupled to the processors. The program instructions may be configured to implement the functions described herein (e.g., functions of various servers and other components that implement the provider network described herein). The various methods shown in the figures and described herein represent exemplary embodiments of the methods. The order of any method may be changed, and various elements may be added, rearranged, combined, omitted, modified, etc.

[0105] The technologies discussed above can be run on one or more computer systems that can interact with various other devices. Figure 17 is a block diagram illustrating exemplary computer systems in various embodiments. For example, computer system 2000 may be configured in different embodiments to implement a compute cluster, a distributed key-value data store, and / or client nodes. Computer system 2000 may be any of various types of devices, including but not limited to personal computer systems, desktop computers, laptop or notebook computers, mainframe computer systems, handheld computers, workstations, network computers, consumer devices, application servers, storage devices, telephones, mobile phones, or any type of computing device in general.

[0106] The computer system 2000 includes one or more processors 2010 (some of which may include multiple cores, which may be single-threaded or multi-threaded) coupled to system memory 2020 via an input / output (I / O) interface 2030. The computer system 2000 further includes a network interface 2040 coupled to the I / O interface 2030. In various embodiments, the computer system 2000 may be a uniprocessor system including one processor 2010, or a multiprocessor system including multiple processors 2010 (e.g., two, four, eight, or another preferred number). A processor 2010 may be any preferred processor capable of executing instructions. For example, in various embodiments, a processor 2010 may be a general-purpose or embedded processor implementing one of various instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPS ISA, or any other preferred ISA. In a multiprocessor system, each of the processors 2010 may, but not necessarily, implement the same ISA in common. Computer system 2000 also includes one or more network communication devices (e.g., network interface 2040) for communicating with other systems and / or components via a communication network (e.g., the Internet, a LAN, etc.). For example, a client application running on system 2000 may use network interface 2040 to communicate with a server application running on a single server or a cluster of servers implementing one or more of the components of the provider network described herein. In another example, an instance of a server application running on computer system 2000 may use network interface 2040 to communicate with another instance of the server application (or another server application) that may be implemented on another computer system (e.g., computer system 2090).

[0107] In the illustrated embodiment, the computer system 2000 also includes one or more persistent storage devices 2060 and / or one or more I / O devices 2080. In various embodiments, the persistent storage device 2060 may correspond to a disk drive, tape drive, solid-state memory, other mass storage device, or any other persistent storage device. The computer system 2000 (or a distributed application or operating system running on it) may, as necessary, store instructions and / or data in the persistent storage device 2060 and, as necessary, retrieve the stored instructions and / or data. For example, in some embodiments, the computer system 2000 may host a storage system server node, and the persistent storage 2060 may include an SSD attached to that server node.

[0108] The computer system 2000 includes one or more system memories 2020 configured to store instructions and data accessible by the processor 2010. In various embodiments, the system memory 2020 may be implemented using any preferred memory technology (e.g., one or more of cache, static random access memory (SRAM), DRAM, RDRAM, EDO RAM, DDR 20 RAM, synchronous dynamic RAM (SDRAM), Rambus RAM, EEPROM, non-volatile / flash type memory, or any other type of memory). The system memory 2020 may include program instructions 2025 that can be executed by the processor 2010 to implement the methods and techniques described herein. In various embodiments, the program instructions 2025 may be encoded in platform native binary, any interpreted language such as Java® bytecode, or any other language such as C / C++, Java®, or any combination thereof. For example, in the illustrated embodiment, the program instructions 2025 include, in a different embodiment, program instructions that can be executed to implement a provider network function. In some embodiments, program instruction 2025 may implement multiple separate clients, server nodes, and / or other components.

[0109] In some embodiments, program instructions 2025 may include instructions executable to implement an operating system (not shown), which may be any of the various operating systems such as UNIX, LINUX®, Solaris®, MacOS®, and Windows®. Any or all of program instructions 2025 may be provided as a computer program product or software which may include a non-temporary computer-readable storage medium storing instructions, which may be used to program a computer system (or other electronic device) to perform processes according to various embodiments, such as various techniques for finding matching code sources according to index and comparative similarity. The non-temporary computer-readable storage medium may include any mechanism for storing information in a form readable by a machine (e.g., a computer) (e.g., software, processing applications). Generally speaking, the non-temporary computer-accessible medium may include a magnetic or optical medium, such as a disk or a computer-readable storage medium or memory medium such as a DVD / CD-ROM, coupled to the computer system 2000 via the I / O interface 2030. Non-temporary computer-readable storage media may also include any volatile or non-volatile media such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc., which may be included in some embodiments of the computer system 2000 as system memory 2020 or other types of memory. In other embodiments, program instructions may be communicated using optical, acoustic, or other forms of propagating signals (e.g., carrier waves, infrared signals, digital signals, etc.) transmitted over a communication medium such as a network and / or wireless link, which may be implemented via a network interface 2040.

[0110] In some embodiments, system memory 2020 may include a datastore 2045 which may be configured as described herein. Generally, system memory 2020 (e.g., datastore 2045 within system memory 2020), persistent storage 2060, and / or remote storage 2070 may store data blocks, copies of data blocks, metadata associated with data blocks and / or their states, configuration information, and / or any other information that can be used when implementing the methods and techniques described herein.

[0111] In one embodiment, the I / O interface 2030 may be configured to coordinate I / O traffic between the processor 2010, the system memory 2020, and any peripheral devices in the system, including via a network interface 2040 or other peripheral interfaces. In some embodiments, the I / O interface 2030 may perform any necessary protocols, timing, or other data conversions to convert data signals from one component (e.g., system memory 2020) into a format suitable for use by another component (e.g., the processor 2010). In some embodiments, the I / O interface 2030 may include support for devices attached via various types of peripheral buses, such as variants of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard. In some embodiments, the functionality of the I / O interface 2030 may be divided into two or more separate components, such as a northbridge and a southbridge. Also, in some embodiments, some or all of the functions of the I / O interface 2030, such as the interface to the system memory 2020, may be directly incorporated into the processor 2010.

[0112] The network interface 2040 may be configured to allow data to be exchanged between the computer system 2000 and other devices attached to the network, such as other computer systems 2090 (which may implement one or more storage system server nodes, database engine head nodes, and / or clients of the database systems described herein). In addition, the network interface 2040 may be configured to allow communication between the computer system 2000 and various I / O devices 2050 and / or remote storage 2070. The input / output devices 2050 may, in some embodiments, include one or more display terminals, keyboards, keypads, touchpads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or reading data by one or more computer systems 2000. Multiple input / output devices 2050 may reside within the computer system 2000 or may be distributed across various nodes of a distributed system including the computer system 2000. In some embodiments, similar input / output devices may be separate from the computer system 2000 and may interact with one or more nodes of a distributed system including the computer system 2000 via wired or wireless connections, such as through the network interface 2040. The network interface 2040 may generally support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11, or another wireless networking standard). However, in various embodiments, the network interface 2040 may support communication over any suitable wired or wireless general data network, such as other types of Ethernet networks. In addition, the network interface 2040 may support communication over telecommunications / telephone networks, such as analog voice networks or digital fiber optic communication networks, communication over storage area networks, such as Fibre Channel SANs, or communication over any other suitable type of network and / or protocol.In various embodiments, the computer system 2000 may include more, fewer, or different components than those shown in Figure 17 (e.g., displays, video cards, audio cards, peripheral devices, ATM interfaces, Ethernet interfaces, Frame Relay interfaces, and other network interfaces).

[0113] It should be noted that any embodiment of the distributed systems described herein, or any of its components, may be implemented as one or more network-based services. For example, a computing cluster within a computing service may present a computing service and / or other type of service employing the distributed computing system described herein to a client as a network-based service. In some embodiments, the network-based service may be implemented by a software and / or hardware system designed to support interoperable machine-to-machine interaction over a network. The network-based service may have an interface described in a machine-readable format, such as Web Services Description Language (WSDL). Other systems may interact with the network-based service in a manner defined by the description of the network-based service's interface. For example, the network-based service may define a variety of actions that other systems may invoke and a specific application programming interface (API) that other systems are expected to conform to when requesting the various actions.

[0114] In various embodiments, network-based services may be requested or invoked through the use of a message containing parameters and / or data associated with a network-based service request. Such a message may be formatted according to a specific markup language, such as Extended Markup Language (XML), and / or encapsulated using a protocol, such as Simple Object Access Protocol (SOAP). To perform a network-based service request, a network-based service client may assemble a message containing the request and transmit the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the network-based service using an Internet-based application layer transport protocol, such as Hypertext Transfer Protocol (HTTP).

[0115] In some embodiments, network-based services may be implemented using Representation State Transfer ("RESTful") techniques rather than message-based techniques. For example, a network-based service implemented according to RESTful techniques may be invoked through parameters contained within HTTP methods such as PUT, GET, or DELETE, rather than being encapsulated within SOAP messages.

[0116] Although the embodiments described above are explained in considerable detail, numerous variations and modifications can be made, as will become apparent to those skilled in the art once the above disclosure is fully understood. The following claims are to be construed as encompassing all such modifications and changes, and therefore the above description is intended to be considered illustrative rather than restrictive.

[0117] Embodiments of this disclosure can be described in light of the following clauses. Clause 1. A system, At least one processor, The system includes memory for storing program instructions, and when the program instructions are executed by at least one processor, it causes at least one processor to implement an integrated development environment, and the integrated development environment is Detecting events that trigger requests for code completion suggestions to be included in code files being edited using a code development system, Sending a request for code completion suggestions to the code generation system, the request being sent at a first time when an event triggering the request for code completion suggestions is detected, and which includes one or more characters immediately preceding the cursor. Receiving a code completion suggestion from a code generation system, wherein the code completion suggestion, at a first time, includes one or more suggested characters to be entered into the code file immediately after the cursor. The number of one or more proposed characters is compared with the corresponding number of one or more actual characters entered into the code file after the first time step, and it is determined that the comparison between the number of one or more proposed characters and the corresponding number of one or more actual characters satisfies one or more verification criteria. A system configured to display code completion suggestions for input into a code file in response to determining that one or more validation criteria are met. Clause 2. The system described in Clause 1, in which events that trigger a request are detected at least in part on the number of keystrokes, the amount of time elapsed since the previous code suggestion, or a combination of the number of keystrokes and the amount of time elapsed since the previous code suggestion. Clause 3. The integrated development environment is further configured to take one or more parts of the code files outside of some previous tokens as additional context, and code completion suggestions are further based on the additional context of one or more parts of the code files outside of some previous tokens, as described in Clause 1 or 2. Clause 4. The code generation system is a system described in any one of Clauses 1 to 3, implemented as part of the code development services provided by the provider network. Article 5. Method, The integrated development environment (IDE) detects events that trigger the acquisition of code completion suggestions to be included in code files being edited using the IDE, The integrated development environment (IDE) triggers the generation of a code completion suggestion based at least partially on one or more characters immediately preceding the cursor, at a first time when an event that triggers a request for a code completion suggestion is detected, and the code completion suggestion triggers the generation of a code completion suggestion based at least partially on one or more characters immediately preceding the cursor, at a first time, on one or more suggested characters to be entered into the code file immediately following the cursor. The integrated development environment compares the number of one or more proposed characters with the corresponding number of one or more actual characters entered into the code file after a first time period, and determines that the comparison between the number of one or more proposed characters and the corresponding number of one or more actual characters satisfies one or more verification criteria. A method comprising: having an integrated development environment display code completion suggestions for input into a code file in response to determining that one or more validation criteria are met. Clause 6. The method of Clause 5, wherein the event triggering the request is detected at least in part on the number of keystrokes, the amount of time elapsed since the previous code suggestion, or a combination of the number of keystrokes and the amount of time elapsed since the previous code suggestion. Clause 7. The method described in Clause 5 or 6, wherein the event triggering the request is detected at least in part on the input of a specific character. Clause 8. The method of any one of Clauses 5 to 7, further comprising discarding a different code proposal generated for inclusion in the code file based on a determination that one or more verification criteria are not met, based on a comparison between further characters entered into the code file and a different code proposal. Clause 9. The method described in any one of Clauses 5-8, further including obtaining one or more portions of the code files outside some prior tokens as additional context, and the code completion proposal is further based on the additional context of the obtained one or more portions of the code files outside some prior tokens. Clause 10. The method of Clause 9, which involves performing one or more regular expression searches to obtain one or more parts of the code file outside of several previous tokens. Clause 11. The method of Clause 5, further comprising taking one or more parts of different code files as additional context, and the code completion suggestion is further based on the additional context of the one or more parts taken from different code files. Clause 12. The method described in any one of Clauses 5 to 11, which includes sending a request for a code completion suggestion to a code development service provided by a provider network, thereby triggering the generation of a code completion suggestion. Clause 13. One or more non-temporary computer-readable storage media for storing program instructions, wherein when program instructions are executed on or across one or more computing devices, the program instructions are stored on one or more computing devices. Detecting events that trigger the acquisition of code completion suggestions to include in code files being edited using a code development system, Obtaining a code completion suggestion at a first time when an event triggering a request for a code completion suggestion is detected, based at least partially on one or more characters immediately preceding the cursor, wherein the code completion suggestion is obtained at a first time, including one or more suggested characters to be entered into the code file immediately following the cursor. The number of one or more proposed characters is compared with the corresponding number of one or more actual characters entered into the code file after the first time step, and it is determined that the comparison between the number of one or more proposed characters and the corresponding number of one or more actual characters satisfies one or more verification criteria. One or more non-temporary computer-readable storage media that, in response to determining that one or more verification criteria are met, display code completion suggestions for input into a code file, and implement the following: Clause 14. One or more non-temporary computer-readable storage media as described in Clause 13, where the event triggering the request is detected at least in part on the number of keystrokes, the amount of time elapsed since the previous code suggestion, or a combination of the number of keystrokes and the amount of time elapsed since the previous code suggestion. Clause 15. One or more non-temporary computer-readable storage media as described in Clause 13 or 14, where the event triggering the request is detected at least in part on the input of a specific character. Clause 16. One or more non-temporary computer-readable storage media as described in any one of Clauses 13 to 15, which store further program instructions causing one or more computing devices to further implement the discarding of different code proposals generated for inclusion in a code file, when a comparison between further characters entered into a code file and different code proposals does not meet one or more verification criteria. Clause 17. One or more non-temporary computer-readable storage media as described in any one of Clauses 13 to 16, which store further program instructions causing one or more computing devices to further implement retrieving one or more portions of code files outside some prior tokens as additional context when executed on or across one or more computing devices, and the code completion proposal further based on the additional context of the retrieved one or more portions of code files outside some prior tokens. Clause 18. When retrieving one or more portions of the code files outside of some prior tokens, a program instruction causes one or more computing devices to implement one or more regular expression searches on one or more non-temporary computer-readable storage media as described in Clause 17. Clause 19. One or more non-temporary computer-readable storage media as described in Clause 13, which store further program instructions causing one or more computing devices to further implement the acquisition of one or more portions of different code files as additional context when executed on or across one or more computing devices, and which further base the code completion proposal on the additional context of the acquired one or more portions of different code files. Clause 20. When obtaining code completion suggestions, a program instruction causes one or more computing devices to implement sending a request to a code development service provided by a provider network for code completion suggestions, using one or more non-temporary computer-readable storage media as described in any one of Clauses 13 to 19. Clause 21. A system, At least one processor, The programming language conversion system comprises a memory that stores program instructions causing at least one processor to implement a programming language conversion system when executed by at least one processor, and the programming language conversion system is The system receives a request to convert an evaluation dataset specified in a first programming language via the interface of a programming language conversion system, wherein different items in the evaluation dataset correspond to different evaluation tests of the code generation system. Converting individual items from different items in a dataset into a second programming language, Convert the function signature of an item in the first programming language to the second programming language. Translate one or more test statements of an item in the first programming language into the second programming language. Following the prompts within the item, a request is sent to the code generation system to generate the body of the function signature translated into the second programming language. Receiving the function signature body in a second programming language from the code generation system, A system configured to store transformed individual items of different items in an evaluation dataset as part of a new evaluation dataset. Clause 22. The second programming language is the system described in Clause 21, as specified in the request for transforming the evaluation dataset. Clause 23. For translating a function signature of an item in a first programming language to a second programming language, the programming language translation system is further configured to identify the respective types in the first programming language of one or more parameters in the function signature that are mapped to corresponding types in the second programming language, as described in Clause 21 or 22. Clause 24. For translating one or more test statements of an item in a first programming language to a second programming language, the programming language translation system is further configured to identify the respective types in the first programming language of one or more parameters in one or more test statements that are mapped to corresponding types in the second programming language, as described in any one of Clauses 21 to 23. Article 25. Method, The process involves receiving an evaluation dataset specified in a first programming language via the interface of a programming language conversion system, wherein different items in the evaluation dataset correspond to different evaluation tests of the code generation system. Converting individual items of different items in a dataset into a second programming language, The programming language conversion system converts the function signature of an item in the first programming language to the second programming language. The programming language conversion system converts one or more test statements of an item in the first programming language into the second programming language. The programming language conversion system generates the body of the converted function signature in the second programming language, following prompts within the item used as input to a machine learning model trained to generate code in the second programming language. A method comprising storing transformed individual items of different items in an evaluation dataset as part of a new evaluation dataset. Clause 26. The method of Clause 25, further comprising receiving a request to convert an evaluation dataset specifying a second programming language. Article 27. Converting individual items of different items in a dataset into a third programming language, The programming language conversion system converts the function signature of an item in the first programming language to the third programming language. The programming language conversion system converts one or more test statements of an item in the first programming language into the third programming language. The programming language conversion system generates the body of a function signature converted in the third programming language, following prompts within an item used as input to a second machine learning model trained to generate code in the third programming language, and The method according to clause 25 or 26, further comprising storing the transformed individual items of different items in a third programming language as part of a second new evaluation dataset. Clause 28. The method described in any one of Clauses 25-27, wherein the translation of a function signature of an item in a first programming language to a second programming language includes identifying the respective types in the first programming language of one or more parameters in the function signature that are mapped to corresponding types in the second programming language. Clause 29. The method described in any one of Clauses 25-28, wherein the translation of one or more test statements of an item in a first programming language to a second programming language includes identifying the respective types in the first programming language of one or more parameters in the test statement that are mapped to corresponding types in the second programming language. Clause 30. A programming language conversion system is implemented as part of the code development services provided by the provider network, and requests to perform conversions are received from clients of the provider network, as described in any one of Clauses 25-29. Clause 31. The method described in any one of Clauses 25-30, further comprising performing natural language conversion on a portion of a prompt in accordance with a second programming language. Clause 32. The method described in any one of Clauses 25-31, which involves sending a request to a code generation system implemented as part of a code development service provided by a provider network, to have the body of a converted function signature generated in a second programming language. Article 33. One or more non-temporary computer-readable storage media for storing program instructions, wherein when program instructions are executed on or across one or more computing devices, the program instructions are stored on one or more computing devices. The process involves receiving an evaluation dataset specified in a first programming language via the interface of a programming language conversion system, wherein different items in the evaluation dataset correspond to different evaluation tests of the code generation system. Converting individual items of different items in a dataset into a second programming language, The programming language conversion system converts the function signature of an item in the first programming language to the second programming language. The programming language conversion system converts one or more test statements of an item in the first programming language into the second programming language. The programming language conversion system generates the body of the converted function signature in the second programming language, following prompts within the item used as input to a machine learning model trained to generate code in the second programming language. One or more non-temporary computer-readable storage media that implement the storage of transformed individual items of different items in an evaluation dataset as part of a new evaluation dataset. Clause 34. One or more non-temporary computer-readable storage media as described in Clause 33, which store further programming instructions causing one or more computing devices to further implement receiving requests to translate evaluation datasets specifying a second programming language when executed on or across one or more computing devices. Clause 35. Store further program instructions, and when program instructions are executed on or across one or more computing devices, on one or more computing devices, Converting individual items of different items in a dataset into a third programming language, The programming language conversion system converts the function signature of an item in the first programming language to the third programming language. The programming language conversion system converts one or more test statements of an item in the first programming language into the third programming language. The programming language conversion system generates the body of a function signature converted in the third programming language, following prompts within an item used as input to a second machine learning model trained to generate code in the third programming language, and One or more non-temporary computer-readable storage media as described in Clause 33 or 34, further enabling the storage of individual converted items from different items of an evaluation dataset in a third programming language as part of a second new evaluation dataset. Clause 36. When translating a function signature of an item in a first programming language to a second programming language, a program instruction causes one or more computing devices to implement the identification of the respective types in the first programming language of one or more parameters in the function signature that are mapped to the corresponding types in the second programming language, Clauses 33-3 5 One or more non-temporary computer-readable storage media as described in any one of the paragraphs. Clause 37. When translating a function signature of an item in a first programming language to a second programming language, a program instruction causes one or more computing devices to implement the identification of the respective types in the first programming language of one or more parameters in a test statement that are mapped to corresponding types in the second programming language, as described in one or more of Clauses 33 to 36. Clause 38. The programming language conversion system is implemented as part of the code development services provided by the provider network, and requests to perform the conversion are received from a client of the provider network on one or more non-temporary computer-readable storage media as described in any one of Clauses 33 to 37. Clause 39. One or more non-temporary computer-readable storage media as described in any one of Clauses 33 to 38, which store further programming instructions causing one or more computing devices to further implement natural language translation of portions of prompts in accordance with a second programming language when executed on or across one or more computing devices. Clause 40. One or more non-temporary computer-readable storage media as described in any one of Clauses 33 to 39, which cause a program instruction to cause one or more computing devices to implement sending a request to a code generation system implemented as part of a code development service provided by a provider network, when generating the body of a converted function signature in a second programming language. Article 41. A system, At least one processor, The code generation system comprises a memory that stores program instructions causing at least one processor to implement a code generation system when executed by at least one processor, and the code generation system is The system receives input programming code and performs the next token prediction for the input programming code, The process involves determining word boundaries for the tokenizer of the input text, where the rightmost boundary contains a subtoken, and the subtoken is used as a prompt suffix. Identifying one or more possible tokens from multiple possible tokens that match the prompt suffix by starting with or ending with the prompt suffix, Filtering the next token prediction according to one or more identified possible tokens, wherein the next token prediction is generated by applying a machine learning model trained to predict the next token in programming code to the rest of the input programming code that does not contain some backtrack tokens corresponding to pretokens, and filtering is performed for one or more iterations after each iteration, removing one or more characters from the left of a subtoken that matches one of the next token predictions, until there are no remaining characters in the subtoken. A system configured to provide the last token prediction of the next set of token predictions as the next token prediction for the input programming code. Clause 42. To identify one or more possible tokens from a plurality of possible tokens that match a partial token, the code generation system is configured to access a tridata structure that stores a plurality of possible tokens, as described in Clause 41. Clause 43. The code generation system, as described in Clause 41 or 42, is configured to access a mask cache storing possible partial tokens in order to identify one or more possible tokens from a plurality of possible tokens that match a partial token. Clause 44. A code generation system is implemented as part of a code development service provided by a provider network, and input programming code is received as part of a request to generate code proposals for code files received from a client of the provider network, as described in any one of Clauses 41 to 43. Article 45. Method, The text generation system receives input text and performs the next token prediction for the input text, The text generation system for the input text determines word boundaries with respect to the tokenizer for the input text, where the rightmost boundary contains a subtoken, and the subtoken is used as a prompt suffix. The text generation system identifies one or more possible tokens from multiple possible tokens that match the prompt suffix by either starting with or ending with the prompt suffix. The text generation system filters the next token prediction according to one or more identified possible tokens, the next token prediction being generated by applying a machine learning model to the rest of the input text that does not contain several backtrack tokens corresponding to pretokens, and the filtering being performed for one or more iterations after each iteration, removing one or more characters from the left of a subtoken that matches one of the next token predictions, until there are no remaining characters in the subtoken. A method comprising providing the last of the following token predictions as the next token prediction by a text generation system. Clause 46. The method of Clause 45, which involves accessing a tridata structure that stores multiple possible tokens, in order to identify one or more possible tokens from a plurality of possible tokens that match a partial token. Clause 47. The trial data structure is used to generate the next token prediction using different machine learning models, Clause 4 5 or 4 6 Methods used. Clause 48. Identifying one or more possible tokens from a plurality of possible tokens that match a partial token, as described in any one of Clauses 45-47, including accessing a mask cache that stores possible partial tokens. Clause 49. The method described in any one of Clauses 45 to 48, further comprising determining the number of backtrack tokens up to the maximum number of backtrack tokens by the text generation system. Clause 50. The method described in any one of Clauses 45-49, wherein input text is received as part of a request to provide the next token prediction, and the next token prediction is provided as a response to the request. Clause 51. A text generation system is implemented as part of an autocomplete application, as described in any one of Clauses 45-50. Clause 52. The text generation system is implemented as part of a code development service provided by a provider network, and the input text is received as part of a request to generate code proposals for code files received from a client of the provider network, as described in any one of Clauses 45 to 51. Article 53. One or more non-temporary computer-readable storage media for storing program instructions, wherein when program instructions are executed on or across one or more computing devices, the program instructions are stored on one or more computing devices. The process involves receiving input text and performing a next token prediction based on that input text, The process involves determining word boundaries for the tokenizer of the input text, where the rightmost boundary contains a subtoken, and the subtoken is used as a prompt suffix. Identifying one or more possible tokens from multiple possible tokens that match the prompt suffix by starting with or ending with the prompt suffix, Filtering the next token prediction according to one or more identified possible tokens, wherein the next token prediction is generated by applying a machine learning model to the rest of the input text that does not contain some backtrack tokens corresponding to pretokens, and filtering is performed for one or more iterations after each iteration, removing one or more characters from the left of a subtoken that matches one of the next token predictions, until there are no remaining characters in the subtoken. One or more non-temporary computer-readable storage media implement the following: providing the last token prediction of the next token prediction as the next token prediction for the input text. Clause 54. When identifying one or more possible tokens from a plurality of possible tokens that match a partial token, a program instruction causes one or more computing devices to implement access to a tridata structure storing a plurality of possible tokens, as described in Clause 53, one or more non-temporary computer-readable storage media. Clause 55. The TriData structure is one or more non-temporary computer-readable storage media as described in Clause 54, used to generate the next token prediction using different machine learning models. Clause 56. When identifying one or more possible tokens from a plurality of possible tokens that match a partial token, a program instruction causes one or more computing devices to implement access to a mask cache storing the possible partial tokens, one or more non-temporary computer-readable storage media as described in any one of Clauses 53 to 55. Clause 57. One or more non-temporary computer-readable storage media as described in any one of Clauses 53 to 56, which store further program instructions causing one or more computing devices to further implement a text generation system determining the number of backtrack tokens up to a maximum number of backtrack tokens when executed on or across one or more computing devices. Clause 58. Input text is received as part of a request to provide the next token prediction, and the next token prediction is provided as a response to the request, on one or more non-temporary computer-readable storage media as described in any one of Clauses 53 to 57. Clause 59. One or more computing devices are one or more non-temporary computer-readable storage media as described in any one of Clauses 53 to 58, implemented as part of an autocomplete application. Clause 60. One or more computing devices implemented as part of a code development service provided by a provider network, and input text received as part of a request to generate code proposals for code files received from a client of the provider network, one or more non-temporary computer-readable storage media as described in any one of Clauses 53 to 59. Article 61. A system, At least one processor, The machine learning system comprises memory that stores program instructions causing at least one processor to implement a machine learning system when executed by at least one processor, and the machine learning system is The system receives programming code to train a machine learning model to predict the next programming code token given an input programming code token, and Analyzing programming code and determining multiple tokens from the programming code, This involves randomly segmenting different tokens from among multiple tokens into multiple sub-tokens, A system configured to train a machine learning model to predict the next programming code token given a programming code token input, using multiple tokens, each containing multiple subtokens, as a training dataset. Clause 62. Random segmentation of different tokens among the tokens is performed in accordance with the subword regularization technique of the system described in Clause 61. Clause 63. The machine learning model is a pre-trained machine learning model, as defined in Clause 61 or 62 of the System. Clause 64. A machine learning system is a system described in any one of Clauses 61-63, implemented as part of a code development service provided by a provider network for training machine learning models to generate code suggestions. Article 65. Method, In a machine learning system, receiving text data to train a machine learning model to predict the next text token given an input text token, The machine learning system determines multiple tokens from text data, The machine learning system randomly segments different tokens from among multiple tokens into multiple subtokens, A method comprising training a machine learning model to predict the next text token given an input text token, using a machine learning system to train a machine learning model to predict the next text token, given an input text token, using a set of multiple tokens, each containing multiple subtokens, as a training dataset. Clause 66. Random segmentation of different tokens among the tokens is performed in accordance with the method of Clause 65, in accordance with subword regularization techniques. Clause 67. A machine learning model is a pre-trained machine learning model, Clause 6 5 or 6 6 Methods used. Clause 68. The method described in any one of Clauses 65-67, wherein the text data is code written in a programming language, and the following text tokens and input text tokens are their respective programming code tokens. Clause 69. The method of Clause 68, wherein determining multiple tokens from text data and randomly segmenting different tokens among the multiple tokens into each of multiple subtokens is performed by a tokenizer, and the tokenizer is applicable to train a second machine learning model in a second programming language to predict the next code token. Clause 70. The method of any one of Clauses 65 to 69, further comprising storing the trained machine learning model in a location specified in the request for training the machine learning model. Clause 71. The method described in any one of Clauses 65-70, further comprising deploying a trained machine learning model as part of an autocomplete application. Clause 72. The method described in any one of Clauses 65-71, where a machine learning system is implemented as part of a code development service provided by a provider network to train a machine learning model for generating code suggestions. Article 73. One or more non-temporary computer-readable storage media for storing program instructions, wherein when program instructions are executed on or across one or more computing devices, the program instructions are stored on one or more computing devices. Receiving text data to train a machine learning model to predict the next text token given an input text token, Determining multiple tokens from text data, This involves randomly segmenting different tokens from among multiple tokens into multiple sub-tokens, One or more non-temporary computer-readable storage media to implement a machine learning model that trains to predict the next text token given an input text token, using multiple tokens, each containing multiple subtokens, as a training dataset. Clause 74. Random segmentation of different tokens among the tokens is performed in accordance with subword regularization techniques on one or more non-temporary computer-readable storage media as described in Clause 73. Clause 75. A machine learning model is a pre-trained machine learning model stored on one or more non-temporary computer-readable storage media as described in Clause 73 or 74. Clause 76. Text data is code written in a programming language, and the following text tokens and input text tokens are their respective programming code tokens, on one or more non-temporary computer-readable storage media as described in any one of Clauses 73 to 75. Clause 77. Determining multiple tokens from text data and randomly segmenting different tokens among the multiple tokens into multiple subtokens is performed by a tokenizer, which is applicable to training a second machine learning model in a second programming language to predict the next code token, on one or more non-temporary computer-readable storage media as described in Clause 76. Clause 78. One or more non-temporary computer-readable storage media as described in any one of Clauses 73 to 77, which store further program instructions causing one or more computing devices to further implement storing the trained machine learning model in a location specified in a request for training a machine learning model when executed on or across one or more computing devices. Clause 79. One or more non-temporary computer-readable storage media as described in any one of Clauses 73 to 78, which store further program instructions causing one or more computing devices to deploy a trained machine learning model as part of an autocomplete application when executed on or across one or more computing devices. Clause 80. A machine learning system is implemented as part of a code development service provided by a provider network for training machine learning models to generate code suggestions, on one or more non-temporary computer-readable storage media as described in any one of Clauses 73 to 79.< / object> < / integer> < / double> < / double> < / integer> < / integer> < / t> < / t> < / t> < / t> < / t> < / t> < / t>

Claims

1. It is a system, At least one processor, The system comprises a memory for storing program instructions, and when the program instructions are executed by the at least one processor, the system causes the at least one processor to implement an integrated development environment, and the integrated development environment is To detect events that trigger requests for code completion suggestions to be included in code files being edited using the aforementioned integrated development environment, To cause the generation of the code completion suggestion based at least in part on one or more characters immediately preceding the cursor, at a first time when the event that triggers the request for the code completion suggestion is detected, the code completion suggestion to cause, at the first time, to include one or more suggested characters for input into the code file immediately following the cursor, A certain number of characters from the one or more proposed characters are compared with a corresponding number of characters from one or more actual characters entered into the code file after the first time period, and it is determined that the comparison between a certain number of characters from the one or more proposed characters and a corresponding number of characters from one or more actual characters satisfies one or more verification criteria. A system configured to display code completion suggestions for input into the code file in response to a determination that one or more of the aforementioned verification criteria are met.

2. The system according to claim 1, wherein the event triggering the request is detected at least in part on the number of keystrokes, the amount of time elapsed since the previous code suggestion, or a combination of the number of keystrokes and the amount of time elapsed since the previous code suggestion.

3. The integrated development environment is further configured to retrieve one or more portions of the code file outside of a context window as context information, the context window being several previous tokens before the cursor, and the code completion suggestions are further based on the context information of the retrieved one or more portions of the code file outside of the context window, according to claim 1.

4. The system according to claim 1, wherein the code completion proposal is generated by a code generation system implemented as part of a code development service provided by a provider network.

5. It is a method, The integrated development environment (IDE) detects events that trigger a request for code completion suggestions to be included in code files being edited using the IDE, The integrated development environment causes the generation of the code completion suggestion, at least partially based on one or more characters immediately preceding the cursor, at a first time when the event that triggers the request for the code completion suggestion is detected, wherein the code completion suggestion causes, at the first time, one or more suggested characters for input into the code file immediately following the cursor. The integrated development environment compares a certain number of characters from the one or more proposed characters with a corresponding number of characters from one or more actual characters entered into the code file after the first time period, and determines that the comparison between the certain number of characters from the one or more proposed characters and the corresponding number of characters from one or more actual characters satisfies one or more verification criteria. A method comprising: determining that one or more of the aforementioned verification criteria are met, and in response, the integrated development environment displays the code completion suggestions for input to the code file.

6. The method according to claim 5, wherein the event triggering the request is detected at least in part on the number of keystrokes, the amount of time elapsed since the previous code suggestion, or a combination of the number of keystrokes and the amount of time elapsed since the previous code suggestion.

7. The method according to claim 5, wherein the event triggering the request is detected at least in part on the input of a specific character.

8. The method of claim 5, further comprising discarding the different code suggestions generated for inclusion in the code file, according to a determination that one or more of the verification criteria are not met, based on a comparison between further characters entered in the code file and different code suggestions.

9. The method of claim 5, further comprising obtaining one or more portions of the code file outside the context window as context information, wherein the context window is some previous tokens before the cursor, and the code completion suggestion is further based on the context information of the obtained one or more portions of the code file outside the context window.

10. The method according to claim 9, wherein obtaining the one or more portions of the code file outside the context window includes performing one or more regular expression searches.

11. The method of claim 5, further comprising obtaining one or more portions of different code files as contextual information, wherein the code completion proposal is further based on the contextual information of the obtained one or more portions of the different code files.

12. The method of claim 5, wherein triggering the generation of the code completion suggestion includes transmitting the request for the code completion suggestion to a code development service provided by a provider network.

13. One or more non-temporary computer-readable storage media for storing program instructions, wherein the program instructions are stored on one or more computing devices, or across one or more computing devices, Detecting events that trigger requests for code completion suggestions to be included in code files being edited using a code development system, Obtaining the code completion suggestion at least partially based on one or more characters immediately preceding the cursor, at a first time when the event triggering the request for the code completion suggestion is detected, wherein the code completion suggestion includes, at the first time, one or more suggested characters for input into the code file immediately following the cursor. A certain number of characters from the one or more proposed characters are compared with a corresponding number of characters from one or more actual characters entered into the code file after the first time period, and it is determined that the comparison between the certain number of characters from the one or more proposed characters and the corresponding number of characters from one or more actual characters satisfies one or more verification criteria. One or more non-temporary computer-readable storage media implement the following: displaying the code completion suggestions for input to the code file in response to determining that one or more of the aforementioned verification criteria are met.

14. One or more non-temporary computer-readable storage media according to claim 13, wherein the event triggering the request is detected at least in part on the number of keystrokes, the amount of time elapsed since the previous code suggestion, or a combination of the number of keystrokes and the amount of time elapsed since the previous code suggestion.

15. The one or more non-temporary computer-readable storage media according to claim 13, wherein, when obtaining the code completion suggestion, the program instruction causes one or more computing devices to implement sending the request for the code completion suggestion to a code development service provided by a provider network.

Images