Verification and provision of actively generated code suggestions

Abstract

Code completion suggestions may be actively obtained and validated. An event that triggers obtaining a code completion suggestion to include in a code file being edited using the integrated development environment may be detected. The code completion suggestion may be obtained. Upon detecting the event that triggers obtaining the code completion suggestion, characters of the code completion suggestion may be compared to characters added to the code file to determine whether the code completion suggestion is valid. A valid code completion suggestion may then be displayed.

Description

Background Art

[0001] Programming languages ​​enable developers, designers, and other users to precisely specify various hardware or software design operations for many different applications. Given the wide variety of programming languages, these developers, designers, and other users may encounter or otherwise use code written in a programming language that the developer may not be familiar with. Code development tools provide developers, designers, and other users with different capabilities to improve code performance and identify errors, which in the exemplary scenario described above can help overcome the problem of developers being unfamiliar with programming languages ​​(or the environment in which programming languages ​​are deployed) so that high-performance code can still be written. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Figure 1 is a logical block diagram illustrating code generation for code development according to some embodiments.

[0003] Figure 2 is a logical block diagram illustrating a provider network that implements different services including code development services according to some embodiments.

[0004] Figure 3 is a logical block diagram illustrating code suggestion processing according to some embodiments.

[0005] Figure 4 is a logical block diagram illustrating code suggestions according to some embodiments.

[0006] Figure 5 is a logical block diagram illustrating an example interface of a development environment according to some embodiments.

[0007] Figure 6 is a logical block diagram illustrating an example of code suggestion development for subword regularization when training a machine learning model according to some embodiments.

[0008] Figure 7 is a logical block diagram illustrating an example of code suggestion development for evaluating data set transformations according to some embodiments.

[0009] Figure 8 is a high-level flow chart illustrating techniques and methods for implementing validation and proactively providing code suggestions according to some embodiments.

[0010] Fig. 9 are example times for detecting, validating, and displaying code completion suggestions according to some embodiments.

[0011] Fig.10 is a high-level flow chart illustrating techniques and methods for implementing generation of code suggestions for input programming code according to some embodiments.

[0012] Fig.11 is a high-level flow chart illustrating techniques and methods for implementing constrained prefix matching for generating next word-gram predictions in accordance with some embodiments.

[0013] Fig.12 is a logic block diagram illustrating different iterations of constrained prefix matching according to some embodiments.

[0014] Fig.13 is a high-level flow chart illustrating techniques and methods for implementing random word-gram partitioning for training a next word-gram prediction model in accordance with some embodiments.

[0015] Fig.14 is a logic block diagram illustrating possible random tokenization of text according to some embodiments.

[0016] Fig.15 is a high-level flow chart illustrating techniques and methods for implementing programmatically generating evaluation data sets for a code generation system in accordance with some embodiments.

[0017] Fig.16A and 16B An example project showing an evaluation dataset before and after transformation according to some embodiments.

[0018] Fig.17 is a block diagram illustrating an example computing system in accordance with some embodiments.

[0019] Although embodiments are described herein by way of example for several embodiments and illustrative figures, those skilled in the art will recognize that embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and detailed descriptions thereof are not intended to limit the embodiments to the specific forms disclosed, but on the contrary, are intended to cover all modifications, equivalents and alternatives falling within the spirit and scope defined by the appended claims. The titles used herein are for organizational purposes only and are not intended to be used to limit the scope of this specification or claims. As used throughout this application, the word "may" is used in an allowed sense (i.e., meaning possible), rather than a mandatory sense (i.e., meaning must). Similarly, the words "include, including and includes" are meant to include but are not limited to. DETAILED DESCRIPTION

[0020] This article describes various techniques for code generation and verification for code development. Complex code development tools can rely on machine learning driven features to help design and develop new applications, systems, or services. In order to truly improve the user experience of these features, the speed and quality of machine learning driven features can depend on various aspects of their development and implementation. One such machine learning driven feature for code development is code suggestions, which can generate and recommend code to developers. Techniques that improve the speed and quality of code suggestions driven by machine learning can improve the user experience and the quality of applications, systems, or services produced using the features.

[0021] Figure 1 1 is a logical block diagram illustrating code generation for code development according to some embodiments. Upon receiving a code file input, an integrated development environment 110 (such as a development application implemented locally on a user device (e.g., a computer or laptop) or a development application hosted as part of a provider network service) can use code suggestion generation 120. As discussed in detail below, the code suggestion process 112 can actively 114 obtain and validate the code suggestions 116 before providing them to the display 104, as described below with respect to Figure 3 , 4 , 8, and 9. In this way, a higher latency programming language suggestion model 122 implemented as part of code suggestion generation 120 can be employed that provides better and more usable code suggestions 116 even though its latency is longer than smaller but less accurate models because active requests can make the apparent latency of code suggestions zero or close to zero while still ensuring through validation that the suggestions are still valid (given the potentially changing context, such as other code file inputs 102) before displaying 104. As described below with respect to Figure 3 Other techniques discussed, such as storing and providing paged results, may also reduce latency in waiting for code suggestions to be displayed 104 .

[0022] Other performance improvements for the use and implementation of code suggestion generation 120 can improve the quality and accuracy of suggestions without affecting model performance. Figure 4 , 11 As discussed in detail in 12, constrained prefix matching techniques can improve suggestions in partial-word scenarios without increasing the time to generate suggestions in a meaningful way.

[0023] Code suggestion development 130 (e.g., training 134 and deployment 136 of programming language suggestion model 122) can have an improved impact on code suggestion performance. For example, as described below with respect to Figure 6 , 13As discussed in detail in 14, techniques for modifying the training dataset to illustrate and train subwords can improve the code suggestions made in those scenarios. In addition, the code suggestion generation 120 and the evaluation of the model 122 can be improved by increasing the number of high-quality evaluation datasets 132, as described below with respect to Figure 7 and 15 -16B, the high-quality evaluation dataset can be generated programmatically from other evaluation datasets.

[0024] Please note that the previous description is not intended to be limiting and is provided only as an example of an integrated development environment, a code suggestion generation system, and a code suggestion development tool. As discussed in detail below, various other embodiments may also implement these techniques.

[0025] The specification next includes a general description of a provider network in which the code development service may be implemented. Various examples of the code development service are then discussed, including different components / modules or arrangements of components / modules that may be used as part of implementing the code development service in the provider network. A number of different methods and techniques are then discussed, some of which are illustrated in the accompanying flow charts. Finally, a description of an example computing system is provided on which various components, modules, systems, devices, and / or nodes may be implemented. Various examples are provided throughout the specification.

[0026] Figure 2 2 is a logical block diagram illustrating a provider network implementing different services, including code development services, according to some embodiments. Provider network 200 (which may be referred to as a "cloud provider network" or simply "cloud" in some embodiments) refers to a network-accessible pool of computing resources (such as computing, storage and networking resources, applications and services), which may be virtualized or bare metal. Provider network 200 can provide convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to accommodate variable loads.

[0027] The provider network 200 can be formed into multiple zones, where a zone is a separate geographic area in which a cloud provider clusters data centers. Each zone may include two or more availability zones connected to each other by a dedicated high-speed network (e.g., a fiber optic communication connection). An availability zone (also referred to as an availability domain, or simply "zone") refers to an isolated fault domain that includes one or more data center facilities, which have separate power, separate networking, and separate cooling separated from facilities in another availability zone. Preferably, the availability zones within a zone are far enough apart from each other so that the same natural disaster should not simultaneously take more than one availability zone offline. Customers can connect to the availability zones of the provider network 200 through a publicly accessible network (e.g., the Internet, a cellular communication network). The zone is connected to a global network that includes a dedicated networking infrastructure (e.g., a fiber optic connection controlled by a cloud provider) that connects each zone to at least one other zone. The provider network 200 can deliver content from a point of presence outside of these zones but networked with these zones by means of edge locations and regional edge cache servers. This division and geographic distribution of computing hardware enables the provider network 200 to provide customers with low-latency resource access with a high degree of fault tolerance and stability on a global scale.

[0028] As described above, the provider network 210 can implement various computing resources or services, such as code development services 210 and other services 230, which can be any other type of network-based services, including various other types of storage services (for example, database services or object storage services), computing services, data processing services, machine learning services, analysis services, communication services, event processing services, visualization services, and security services not shown.

[0029] In various embodiments, Figure 2 The components shown may be implemented directly in computer hardware, as instructions executable directly or indirectly by computer hardware (e.g., a microprocessor or computer system), or using a combination of these technologies. Figure 2 The components of can be implemented by a system including multiple computing nodes (or simply referred to as nodes), each of which can be similar to Fig.17 1 and described below. In various embodiments, the functionality of a given system or service component (e.g., a component of code development service 210) may be implemented by a specific node or may be distributed across several nodes. In some embodiments, a given node may implement the functionality of more than one service system component (e.g., more than one data storage component).

[0030] In some embodiments, code development service 210 may be implemented by provider network 200. Code development service 210 may implement various features to write code for different systems, applications, or devices, providing features for suggesting, identifying, reviewing, building, and deploying code. For example, code development service 210 may implement development environment 211. Code development environment 211 may provide various code entry tools (e.g., text-based, chart / graphic application development) to specify, call, or otherwise write (or cause to be written) code for different hardware or software applications.

[0031] The code development service 210 may implement code suggestion delivery 214, which may implement various computing resources to host and implement code suggestions 213 in a scalable manner to deliver on-demand code suggestions across a large number of clients using a highly driven machine learning model for high-quality code suggestion results. For example, the code suggestion delivery 214 may implement workload balancing features and request management features to process and return code suggestions in a timely manner, thereby providing real-time code suggestions to the code suggestion process 220 (within or without the provider network 200) with little or no apparent delay.

[0032] To avoid having the development environment wait to send multiple code suggestions in one communication, in some embodiments, code suggestion delivery 214 can implement a paging feature for code suggestions to allow multiple code suggestions to be delivered from the host or other computing resource implementing the code suggestions 213 to the recipient development environments 219 and 211 over time through multiple communications. In this way, valid code suggestions can be made and presented, and then updated as more suggestions are received. Such techniques provide a simulated streaming experience without actually needing to support bidirectional streaming in the development environment. In this way, the benefits of fast delivery and updating of code suggestions can be provided without introducing additional requirements for the development environment, which may not necessarily be maintained by the provider network 200 operator.

[0033] To implement paging, code suggestions may be stored in the service 210 as they are generated and then returned in multiple exchanges by utilizing paging tokens accompanying requests for code suggestions to allow additional code suggestions to be retrieved from storage and sent back to the development environment 219 or 211.

[0034] In various embodiments, code suggestions 213 can generate code suggestions based on text input in development environment 211 or 219 (e.g., using a plug-in or other connection that can provide real-time analysis and suggestions of code as code is entered into development environment 211 or 219), as described below with respect to Figure 3Discussed in detail. Code suggestions 213 can use a generative model, a machine learning model, such as a generative pre-trained transformer (GPT), that is trained to generate code suggestions. Generative models are typically trained on large amounts of data for a specific task. In the case of generating code recommendations, this corpus (e.g., from the code suggestion code base 215 or other code base used to train the generative model) can include code bases or fragments from various sources. Depending on the source code or owner, the code may be subject to certain licenses during any use or reproduction. Since generative models can sometimes reproduce matches with training data verbatim or nearly verbatim, it may also be necessary to provide metadata for determining the original source attributes as part of the suggestion. Code suggestion metadata (not shown) can provide the ability to provide metadata for code suggestions that can be provided.

[0035] The code development service 210 may implement (or access) a code repository 215. The code repository 215 may store various code files, objects, or other code that may interact with various other features of the code development service 210 (e.g., a development environment 211 for writing, building, compiling, and / or testing code). In some embodiments, the code repository 215 may implement various version and / or other access controls to track and / or maintain consistent versions of code collections for various development projects. In some embodiments, the code repository may be stored or implemented external to the provider network 200 (e.g., hosted in a private network or other location).

[0036] The code development service 210 may implement interfaces to access and / or utilize various features of the code development service 210. Such interfaces may include various types of interfaces, such as command line interfaces, graphical user interfaces, and / or programming interfaces (e.g., application programming interfaces (APIs)) to perform requested operations, including operations of the development environment 211. An API refers to an interface and / or communication protocol between a client and a server such that if a client issues a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In the context of a cloud provider network, an API provides a gateway for clients to access cloud infrastructure by allowing them to obtain data from the cloud provider network or to initiate actions within the cloud provider network, thereby enabling the development of applications that interact with resources and services hosted in the cloud provider network. An API may also enable different services of the cloud provider network to exchange data with each other.

[0037] In general, client 250 may include any type of client that is configurable to submit network-based requests to provider network 200 via network 260, including requests for services (e.g., requests for code searches or suggestions, etc.). For example, client 250 may include a suitable version of a web browser, or may include a plug-in module or other type of code module that can be executed as an extension of an execution environment provided by a web browser or executed within the execution environment. Alternatively, client 250 may include an application (or its user interface), a media application, an office application, or any other application that can utilize resources in provider network 200 to implement various applications. In some embodiments, such applications may include sufficient protocol support (e.g., for a suitable version of hypertext transfer protocol (HTTP)) for generating and processing network-based service requests without having to implement full browser support for all types of network-based data. That is, client 250 may be an application that interacts directly with provider network 200. In some embodiments, client 250 may generate network-based service requests based on a network-based service architecture of a representative state transfer (REST) ​​style, a network-based service architecture based on a document or message, or another suitable network-based service architecture.

[0038] In some embodiments, client 250 can provide access to provider network 200 to other applications in a transparent manner to these applications. For example, client 250 can be integrated with code development services 210. However, the operating system or file system can present different storage interfaces to the application program (such as the conventional file system hierarchy of files, directories and / or folders). In such embodiments, it may not be necessary to modify the application program to utilize the storage system service model. Instead, the details of docking with the data storage service can be coordinated by the client 250 and the operating system or file system on behalf of the application program executed in the operating system environment.

[0039] Client 250 can transmit network-based service request to provider network 200 through network 260, and receive response from the provider network. In various embodiments, network 260 can cover any suitable combination of networking hardware and protocol necessary for establishing network-based communication between client 250 and provider network 200. For example, network 260 can generally cover various telecommunication networks and service providers that implement the Internet together. Network 260 can also include private networks, such as local area network (LAN) or wide area network (WAN) and public or private wireless networks. For example, both given client 250 and provider network 200 can be provided in enterprises with their own internal networks respectively. In such embodiments, network 260 can include hardware (e.g., modem, router, switch, load balancer, proxy server, etc.) and software (e.g., protocol stack, accounting software, firewall / security software, etc.) necessary for establishing networking link between given client 250 and the Internet and between the Internet and provider network 200. It should be noted that in some embodiments, client 250 can communicate with provider network 200 using private network instead of public Internet.

[0040] In some embodiments, the provider network 200 may include the hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall / security software, etc.) necessary to establish networking links between different components of the provider network 200, such as virtualization hosts, control plane components, and external networks 260 (e.g., the Internet). In some embodiments, the provider network 200 may employ Internet Protocol (IP) tunneling technology to provide an overlay network through which encapsulated packets may be tunneled through an internal network. The IP tunneling technology may provide a mapping and encapsulation system for creating an overlay network, and may provide separate namespaces for the overlay layer and the internal network layer. Packets in the overlay layer may be checked against a mapping directory to determine what their tunnel destination should be. The IP tunneling technology provides a virtual network topology; the interface presented to the client 250 may be attached to the overlay network so that when the client 250 provides the IP address to which it wants to send a packet, the IP address operates in the virtual space by communicating with a mapping service that knows where the IP overlay address is.

[0041] Perceived delays in code suggestions may reduce the utilization of code suggestions as a feature. For example, if a user must wait for a noticeable period of time after requesting code suggestions, the user workflow may be interrupted. To eliminate perceived delays, code suggestions may be pre-fetched. However, since the input context may have changed since the code suggestions were requested, techniques for validating proactively obtained code completion suggestions may be implemented, which ensure that the recommendations given are no longer consistent with the current state of the code. Figure 3 is a logical block diagram illustrating code suggestion processing according to some embodiments.

[0042] The code suggestion processing 220 can implement automatic suggestion event detection 310 that can evaluate keystrokes 342, elapsed time, special keys, or user-specific data to detect events. In various embodiments, this information can be maintained as part of a user-specific state that can be updated or reset when a code suggestion request is submitted. For example, keystrokes, elapsed time, or other measurements can be reset. Special keys can also be triggering events (and can also be evaluated in combination with other criteria (such as elapsed time)). For example, an event that triggers obtaining code completion suggestions can include entering "{""["" ("":" "ENTER key" or "TAB key") and an elapsed time threshold. In some embodiments, the automatic suggestion event detection 310 can use client-specific events, such as entering specific keys or characters in a client-specific mode (or configured / described by the client in the request to configure the suggestion processing 220).

[0043] Code suggestion request execution 320 may handle the formation, assembly, sending, and processing of responses from code suggestions 213, including sending requests 322 to obtain code completion suggestions and processing returned code suggestions 324. For example, code suggestion request execution 320 may obtain a context window of tokens (e.g., the N previous tokens before the cursor) from file state 340, as indicated at 344. In some embodiments, file and other context information as provided by file and other context extraction 350 may be sent.

[0044] File and other context extraction 350 can utilize different techniques to obtain file and other context information outside the context window (e.g., outside the N previous tokens). For example, file context can be obtained from the same file as the code suggestions being generated for inclusion therein. Information that can be obtained for file context can include boundaries in the current scope (e.g., code and comments limited by the current function to provide local context), class-level information (including class declarations, class constructors (e.g., __init__ functions)), and function-level information about all other public or protected methods defined by the class, function-level information, including all functions declared on both sides of the cursor in the current file. In some embodiments, signatures, docstrings, and return statements and / or variable-level information can be extracted, including all previous variable declarations that are visible to the current generation focus.

[0045] Other contexts that can be extracted at 350 can be intra-project contexts. In modern code development, classes and functions are typically defined in hierarchical files. Simple retrospective contexts do not include information outside the current file, which can create certain scenarios where machine learning models are unlikely to generate correct code. Because many code files use imported classes / functions / variables, adding this context can significantly improve code generation performance. Therefore, in some embodiments, an intra-project context can be added, where all classes, functions, and variables imported from the same project are used to obtain code completion suggestions.

[0046] The other context that can be extracted at 350 can be an out-of-project context. Out-of-project context can refer to categories / functions / variables imported into the current file from other packages. This may affect the quality of suggestions when the imported packages are in a zero-shot setting (e.g., when the pre-trained model has no prior knowledge about the packages). Therefore, the other context can be obtained by scanning the out-of-project context of packages that are not included in the pre-trained data and including the corresponding categories / functions / variables as context in the request.

[0047] File and other context extraction 350 can perform regular expression based searches (e.g., searching for keywords such as "import") and extraction to obtain the various types of context discussed above. In some embodiments, parsing based extraction (e.g., by generating a symbol tree or other parsing graph of the code to obtain other context information) can be used.

[0048] The code suggestion request execution 320 can interact with the code suggestions provided in a paginated form. For example, a response 324 to a request for code suggestions can include a paginated token indicating that additional suggestions can be retrieved. The code suggestion request execution 320 can still continue to verify and provide suggestions through the code suggestion verification 330, while also submitting a subsequent request 322 with a paginated token to obtain additional code suggestion results, which can then be returned, verified, and provided. In this way, multiple suggestions can be made, allowing different execution times of the code suggestions to be generated, including potentially better code suggestions that may be provided when the user reviews the initial suggestion.

[0049] In some embodiments, the file status 340 may provide information to various stages and may include both the code file and its associated metadata. The file status 340 may also provide information for the code suggestion verification 330, such as the current character before the cursor.

[0050] Code suggestion validation 330 can validate received code suggestions before providing them for display. For example, code suggestion validation can use one or more validation criteria to determine whether the added characters of the code suggestion are a match or a near match of the character before the cursor (and are added after the time the code suggestion request was made). As indicated at 352, valid code suggestions can be provided for display. In some embodiments, acceptance (or rejection) of the suggestions can be received as indicated at 354 and passed to or included in the file state 340 as indicated at 356.

[0051] Code suggestion validation 330 may identify and display 348 valid coded suggestions and process acceptance or rejection of the suggestions,

[0052] Code suggestions 213 may be implemented in different scenarios to provide various code suggestions. Figure 4 4 is a logical block diagram illustrating code suggestions according to some embodiments. A code suggestion request 401 may be received at a tokenizer 410. For example, various different tokenizers or tokenization techniques may be used. Tokens may be individual words in a sentence, including the leading space character "[space] word" before a word as a token. Punctuation marks, empty spaces, carriage returns, or various other characters may also be grouped into or individually considered tokens.

[0053] The programming language word prediction model 420 can use the provided code 422 as well as other context 424, such as file context outside the word window or other files that can be obtained using techniques such as regular expressions or parsing, as described above with respect to Figure 3Programming language specific word prediction models 420 can be used (e.g., model A for language A, model B for language B, etc.). Each of them can be trained with code in the corresponding programming language and other contexts to generate recommendations. Programming language word prediction models 420 can also be trained on other context information (e.g., the same file, the same project, or other contexts), as described above with respect to Figure 3 discussed.

[0054] In some embodiments, the code development service 210 may support a custom programming language model. For example, training data or code data from a user's specific code base may be provided to train a custom programming language model so that it can be used for code suggestions.

[0055] The predictions may be provided to a selection 430 where one prediction may be selected based on the confidence score to be provided as a code suggestion. In some embodiments, multiple predictions may be provided in a paged format or other multi-result format, as described above with respect to Figure 3 discussed.

[0056] One scenario where machine learning models that generate text recommendations can fail is when the input has partial words (like "Syst"). In these scenarios, machine learning models tend to provide poor predictions and, therefore, poor recommendations (e.g., generating nonsense data or generating illogical content). This happens because the model only considers word lemmas as input units. To overcome this scenario, backtrack to the last complete lemma and constrain the generation to match the prompt suffix (here "Syst"). As discussed below, constraining the generation helps improve the accuracy of the subword data metric without compromising the gain on the general evaluation set.

[0057] Given a string hint, inconsistencies from normal decoding may be caused by a suffix of the hint, which may appear with a subword that is not a complete word. Matching of the input string suffix can be performed with all available word-grams that start with the suffix or with which the suffix starts. In some embodiments, matching is performed efficiently using a character dictionary tree data structure (e.g., using a local Pytorch array as a fast cascaded node list). Based on a list of matching word-grams, other word-grams can be masked during the next word-gram prediction to ensure that the generation will match the suffix character by character. Further latency optimization can be achieved by maintaining a Boolean mask to cache very frequent suffixes (such as a single space). For each step of matching with the suffix, after each word-gram generation step, the matching word-gram (by character) is removed from the left side of the suffix and a constrained generation is performed until the suffix is ​​an empty string. In some embodiments, partial tokens (e.g., suffixes) are determined by using the same pre-word splitting as in the pre-word strategy of tokenizer 410, which uses word boundaries to perform the splitting - this allows efficient backtracking for character matching because it is known with certainty that any partial token cannot cross a pre-word boundary.

[0058] Constrained prediction generation 440 may be implemented as described below with respect to Fig.11 and 12 These techniques are discussed in more detail to provide code suggestions. For example, constrained prediction generation 440 can be a feature that is enabled (or disabled) for code suggestions. In some embodiments, constrained prediction 440 can be configured (e.g., via a suggestion request) to utilize a specified maximum number of backtracking tokens (which can be dynamically determined for code suggestions, up to a maximum). Constrained prediction generation 440 can receive tokenized input data from tokenizer 410, and then determine backtracking tokens and partial tokens.

[0059] In various embodiments, constrained prediction generation 440 can then identify one or more possible word-grams that match the partial word-grams that can be identified from the possible word-grams. For example, the possible word-grams can be a vocabulary of a language (e.g., a programming or human language) that can have different predictable words. A match can be identified when the partial word-gram matches at the beginning or end of the possible word-gram (e.g., the possible word-gram matches when the partial word-gram [SYS] matches the beginning of the possible word-gram [SYS* or the end of the possible word-gram *SYS]). The matching can be performed using a dictionary tree structure as discussed above and below. A cache of ordinary matches can also be utilized.

[0060] Constrained prediction generation 440 can then perform one or more iterations of next word-gram prediction, filtering the identified possible matches, evaluating the remaining next word-gram predictions using the partial word-grams, and subtracting matching characters from the partial word-grams until all characters in the partial word-grams have been matched. Code suggestions can then be provided (e.g., to prediction selection 430 which can forward or send only the next word-gram predictions as code suggestions 401).

[0061] Figure 5 5 is a logic block diagram showing an example interface of a development environment according to some embodiments. The integrated development environment interface 500 can be as follows: Figure 2 The implementation on the client of the code development service 210 as depicted, or as Figure 2 5. The IDE interface 500 may be hosted as part of the code development service 210. The IDE interface 500 may implement a code editor 510 (e.g., a text editor) that may allow a user to enter code in a programming language. The code suggestion feature 213 of the code development service 210 may analyze the entered characters to determine code suggestions 520 that may be displayed and added, as indicated at 522. Although not shown, various other information about the suggestions may be displayed, such as the source of the code, license information for the code, and / or various other code metadata (e.g., a style guide).

[0062] Code suggestions and other next word or token prediction techniques may encounter scenarios where a partial token is the nearest contextual input for a code suggestion (or other next word / token prediction). Consider the following example scenario, where a code suggestion is made after a cursor in the following Java code instance, where the cursor is represented as " <t>”:

[0063] GetRecordsResult result=streamClient.getRecords(streamName);

[0064] while(result.getNextRecordMarker()!=null){

[0065] / / (2)

[0066] System.out.println(result.getNextRecordMarker());

[0067] result=streamClient.getRecords(streamName);

[0068] S <t><-current cursor

[0069] Here the expected and most likely guess after the cursor should be System, which is a very common module in Java and which also appears in the context above. However, because "S" is a partial token, the code suggestions provided before the current cursor may not handle the partial token well. Consider the following example input, where the suggested code is underlined after the cursor:

[0070] / / Input->S

[0071] S <t> leepersleeper= new Sleeper(1000) ;

[0072] / / Input->Sy

[0073] Yes <t> strace.endSection() ;

[0074] / / Input->Sys

[0075] Sys <t> out.println(result.getNex tRecordMarker()) ;

[0076] / / Input->Syst

[0077] Syst <t> en.sleep(1000) ;

[0078] / / Input->Syste

[0079] Syste <t> .sleep(1000) ;

[0080] In the above example where a longer partial token is created, the possible suggestions may still not provide the expected "System" result. This may happen because the machine learning model used to generate the suggestions was not trained on partial tokens such as "S" or "Sy", etc.

[0081] This lack of training can occur because the tokenizer breaks down the training dataset used to generate the proposed machine learning model. Consider the following example token generated from the input text (to the left of "->"):

[0082] System->['System']

[0083] Sleeper->['S','le','eper'] / / Prefix S<>

[0084] Systrace->['Sy','str','ace'] / / Prefix Sy<>

[0085] Sysout->['Sys','out'] / / Prefix Sys<>

[0086] Systen->['S','yst','en'] / / Prefix Syst

[0087] Syste->['S','yst','e'] / / Prefix Syste

[0088] To address this problem, a random partitioning of tokens can be applied to the training dataset to train or fine-tune a machine learning model for code (or other text) suggestions. For example, a code text instance can be:

[0089] System.out.prinln("test");

[0090] If you tokenize it as is, it will look like the following:

[0091] ['System','.','out','.','pr','in','ln','("','test','");']

[0092] Instead of doing this, you can insert a random split into the sentence, breaking it into 2 parts:

[0093] 'System.out.prinln("test");'->['S','ystem.out.prinln("test");']

[0094] Each fragment is then lemmatized individually and then concatenated, which will provide the following lemmatization result:

[0095] ['S','y','st','em','.','out','.','pr','in','ln','("','test','");']

[0096] It can be used as part of the training dataset for pre-training or continuous fine-tuning.

[0097] In various embodiments, implementing randomized partitioning of word-grams can train a machine learning model on how to construct various configurations of partial word-grams relative to the original word-grams (e.g., "Sy" to "System"). Such techniques for training machine learning models can significantly improve the accuracy of partial word-gram completion without compromising full word-gram prediction scenarios. Furthermore, these techniques can be implemented without slowing down the machine learning model at the inference stage.

[0098] Figure 6 6 is a logical block diagram illustrating an example of code suggestion development for subword regularization when training a machine learning model according to some embodiments. The random subword tokenizer 610 and model training can be implemented as a tool, system, or feature of the code suggestion development 217. In other embodiments, a separate training system, application, or service (such as implemented as Figure 2 A machine learning service that is part of a provider network 200 in the provider network 200 may implement these techniques and the following Fig.13 and 14 The technologies discussed.

[0099] The random subword segmenter 610 can obtain the training data 602 and randomly divide it, as described below for model training 620 Fig.13 and Fig.14 discussed. For example, a request to generate partial word-gram optimized training data may be received, the request may specify a storage location or other information describing a source training data set including text data. According to some embodiments, a plurality of word-grams may be determined from text data by a random subword tokenizer. For example, a variety of different tokenizers may be applied to generate word-grams from input text. For example, word-grams may be individual words in a sentence, including a space character "[space] word" before the word as a word-gram. Punctuation marks, empty spaces, carriage returns, or various other characters may also be grouped as or individually considered as word-grams.

[0100] In some embodiments, the random subword tokenizer 610 can randomly select different tokens from a plurality of tokens to be divided into corresponding sub-tokens. For example, a subword regularization technique can be performed to sample or identify different tokens that are not optimally divided. Such techniques can include randomly selecting tokens from tokens determined for text data (e.g., based on a configurable variable or parameter that can be indicated as a percentage value, wherein the percentage value indicates the likelihood that any one token will be selected for random division). If a token is selected, the token, rather than the source token from which it is generated, can be randomly divided into sub-token components that are considered to be tokens.

[0101] Model training 620 can implement various machine learning training frameworks that perform machine learning jobs, applications, or programs on the initial model 604 using the training data set generated by the random subword tokenizer 610. Note that training can also be performed to train the model from scratch or to train in view of non-regularized checkpoints. For example, the initial model can include various neural network-based machine learning models that have been pre-trained and are therefore provided for fine-tuning, or can be a new model that has not been pre-trained. Various different hyperparameters or other configurations for model training can be specified as part of a training job or request and used to perform training on the initial model. Once completed, model training 620 can provide a subword regularized trained model, as indicated at 606.

[0102] Another tool, system or feature of code suggestion development 217 can be a programming language conversion 710 that can convert a source evaluation data set of a programming language into another programming language. It is very time-consuming to create a high-quality evaluation data set and the annotator usually needs to consume a lot of time and energy. It is also true for the function completion evaluation set based on execution. In various embodiments, from source programming languages ​​such as Python to programming test conversion tools (and therefore value-oriented conversions) of other target programming languages ​​that are suitable for performing tests of correctness evaluation based on the return value of a function with a benchmark truth value. These embodiments can be used to convert many test cases, which helps to reduce annotation time and increase the quantity of the evaluation data set of other code generation systems for building and testing many different languages. These technologies can be widely applicable and can be used to convert existing data sets such as MBPP (the most basic Python programming) into Javascript, Java, Typescript, Ruby, Go, C# or any other programming language for which a conversion rule set is generated.

[0103] In some embodiments, the conversion process starts with inferring the types of function arguments (which can be completed by inspecting the argument values ​​in each test case). There is a type mapping from python to each language, such as "list" to "Arraylist" or "dictionary" to "Hash Map" in Java. The values ​​of different test cases can have different types, so the common superclass of all observed types of each argument can be inferred based on the type hierarchy. Since there can also be many levels of types (due to containers such as lists or sets), these types can be recursively inferred to be consistent at each level. For example, 'list of lists' and 'list of objects' have a common type of 'list of objects'. The type of the expected return value can also be inferred by inspecting the expected return value in the test case, which will also match the value of the executed function with the given input of the test case.

[0104] In addition to types, arguments and return values ​​are converted from the source programming language to the target programming language by generating strings representing the objects of the target language, which can be parsed by its corresponding interpreter / compiler. For example, [1,2] in Python is converted to 'Arrays.aslist(1,2)' or {1:2,3:["foo","bar"]} in Python is converted to 'new HashMap(){{put(1,2);put(3,Arrays.aslist("foo","bar"))' with recursive support for any nested structure.

[0105] In some embodiments, for test case conversion, all information about return types and arguments / expected return values ​​may be put together to build code representing the input / output objects of the target programming language with appropriate equality comparers.

[0106] Additionally, in some embodiments, the conversion of source programming language hint strings (including function signatures and documentation strings containing input / output examples) can be converted into hint strings for other target programming languages. The style of function signatures can be mimicked in each language as well as appropriate return / independent variable types (if applicable), and conversion of function / independent variable / class names to be style appropriate (camel case or pascal case). Documentation strings can be formatted such that input and output look as close as possible to the input / output format of a particular language.

[0107] Figure 7 is a logical block diagram illustrating an example of code suggestion development for evaluating data set transformation according to some embodiments. Figure 15-16B As discussed in detail, programming language conversion 710 can convert a given evaluation data set from a source programming language to a target programming language. For example, programming language conversion 710 can receive a conversion request 702. The conversion request 702 can specify a source programming language and a target programming language of the evaluation data set, as well as a storage location or other access information of the source evaluation data set, such as a source evaluation data set of programming language A 772, and a target storage location, format, and / or other access information for generating a new evaluation data set, such as a storage location of a new evaluation data set of programming language B 774.

[0108] Programming language transformation 710 may utilize different transformation techniques for different portions of the items in the evaluation data set, such as techniques for inferring or mapping types of function signatures (e.g., recursively) at 720, test statement transformation 730, and natural language transformation 750. Each of these features may utilize specific transformation rules, mappings, and / or type hierarchies for a specified source programming language and target programming language.

[0109] Function signature conversion 720 can identify function signatures in source 772 by parsing items of an evaluation data set according to a parser or rule set of a first (source) programming language to locate function signatures. For example, in the python programming language, a search (e.g., a regular expression search) can be performed on "def" to locate the function signature, which can also be defined by various other symbols (e.g., can include independent variables within parentheses). Once the function signature is located, different techniques can be performed to determine the type of the corresponding independent variables or parameters of the function. For example, a test case for the function can identify the value of the independent variable. To complete the conversion, one or mapping rules specific to the function signature conversion from the source programming language to the target programming language can be applied.

[0110] Test statement transformation 730 may transform the test statement using knowledge determined as part of transforming function signature 720. For example, the argument format of the function signature in the source may be used to extract various test values ​​from the source test statement and insert them into the target programming language version of the test, where the various test values ​​may be available as template test statements that accept arguments and trigger an error or other indication when the test statement fails.

[0111] The natural language transformation 750 can be implemented as part of converting the prompts from the source 772 into the target programming language evaluation data set 774. For example, the transformation of the prompts can include changing features, such as symbols used to indicate code comments (e.g., non-executable statements in the code), such as changing from """"" to " / **". The transformation of the prompts can also include changing natural language statements to replace source programming language terms with target programming language terms, such as changing "write a function in Python" to "write a function in Java", to ... or changes between terms, such as "none" to "empty". In some scenarios, the transformation rules can remove unnecessary or non-convertible source programming language specific statements.

[0112] The function body in the target programming language may be generated by sending a request from 740 to code suggestion 213 which may receive the request 704 and return the generated code 706. In some embodiments, the request may include the converted hints of the test project and the converted function signatures.

[0113] The above discussion on the code development service has been given for an example Figure 2-7 Examples of validation and proactive provision of code suggestions are discussed. Various other types of code development tools, systems, or applications may implement these techniques. Figure 8 is a high-level flow chart illustrating techniques and methods for implementing validation and proactively providing code suggestions according to some embodiments. These techniques and the following description of Figure 9-16B The techniques discussed above can be used as described above with respect to Figure 2-7 The various components of the provider network described may be implemented by other types of code development tools or other applications or systems.

[0114] In some embodiments, as indicated at 810, an event that triggers obtaining code completion suggestions to be included in a code file being edited using an integrated development environment can be detected. In some embodiments, the event that triggers obtaining code completion suggestions can be based on one or more criteria. For example, a keystroke count can be maintained (since the last code completion suggestion request was issued). This keystroke count itself can be an event that is triggered when the number of keystroke counts exceeds a threshold. In some embodiments, other criteria can be considered. For example, the time elapsed since the last trigger can also be used, which can obtain code suggestions after a certain period of time has passed since the last recommendation was made. In some embodiments, a combination of criteria (e.g., keystrokes and elapsed time) can be used. In some embodiments, event triggering can be user or client specific based on heuristics such as entering or using specific characters or keys (e.g., after selecting the TAB key for indentation).

[0115] In some embodiments, as indicated at 820, generation of code completion suggestions may be caused. In some embodiments, the code completion suggestions may be based on the character immediately before a cursor at a first time that an event that triggers a request for code completion suggestions is detected, and wherein the code completion suggestions include suggested characters that were entered into the code file immediately after the cursor at the first time. In some embodiments, the code suggestions may be implemented and executed locally (e.g., by a local subsystem). In some embodiments, the code suggestions may be generated remotely (e.g., as a Figure 2 Features of the code development service 210 in ).

[0116] As indicated at 830, it may be determined whether a comparison of the suggested number of characters to the actual number of characters entered into the code file after the first time satisfies one or more validation criteria. For example, the validation criteria may be an exact match, as described below with respect to Fig. 9 In other embodiments, the validation criteria may allow for fuzzy or approximate matches (e.g., matching 3 out of 4 characters). If no match is found, the code suggestion may be discarded as indicated at 850. Otherwise, the code suggestion may be displayed as indicated at 840.

[0117] In some embodiments, the user can accept or reject the code completion suggestion, which can itself trigger additional code completion suggestion recommendations.

[0118] Fig. 9 is an example time for detecting, verifying, and displaying code completion suggestions according to some embodiments. At time T1, the input code 910 in the editor and where the cursor is are shown. An event that triggers getting code suggestions is detected. At time T2, the input code 920 has changed with the addition of additional characters, as indicated by the moving cursor. The code suggestion 930 can be verified by comparing multiple (e.g., 4) characters with the additional characters "temp" at T2.

[0119] In various embodiments, code suggestions for programming languages ​​may be provided through a code suggestion feature of a service such as code development service 210 or as a standalone code generation system. Many of the techniques discussed above and below for improving performance of various stages of code suggestions may be integrated with code suggestions generated using techniques similar to those of FIG. 10 . Fig.10 is a high-level flow chart illustrating techniques and methods for implementing generation of code suggestions for input programming code according to some embodiments.

[0120] In some embodiments, as indicated at 1010, a request to generate code suggestions for input programming code may be received. For example, in some embodiments, the code suggestion request may be provided as described below with respect to Figure 3 , 8 9, or as a manual request for recommendations. In some embodiments, the request can be used to perform various techniques, such as generating code suggestions to provide transformed function bodies for evaluating data set transformations, as described below with respect to Figure 15-16B discussed.

[0121] In some embodiments, as indicated at 1020, word-grams may be determined from input programming code. Various word-gramming techniques discussed above and below may also be performed. For example, various different word segmenters or word-gramming techniques may be used. Word-grams may be individual words in a sentence, including the leading space character "[space] word" before a word as a word-gram. Punctuation marks, empty spaces, carriage returns, or various other characters may also be grouped as or individually considered as word-grams.

[0122] In some embodiments, as indicated at 1030, a machine learning model trained to generate next word-gram predictions for a programming language corresponding to the programming language of the input programming code can be applied to the word-grams of the input programming code to generate next word-gram predictions for the input programming code. This machine learning model can be trained using randomized word-gram partitioning, as described above with respect to Figure 6 And the following about Fig.13 and 14 Various different types of machine learning models can be used, such as Generative Pre-Trained Transformers (GPT), sequence-to-sequence models, or other neural network-based models such as Long Short-Term Memory (LSTM).

[0123] In some embodiments, one of the next word-gram predictions may be selected based on the corresponding confidence score to be returned as a code suggestion for input into the programming code, as indicated at 1040. In some embodiments, multiple recommendations may be generated and multiple recommendations including multiple recommendations (a number of recommendations, e.g., top 3 recommendations stored based on confidence) may be provided.

[0124] As mentioned above about Figure 1 , 5 As discussed in 6, some word-meta scenarios may cause machine learning models to produce incorrect text suggestions. Although some techniques can be applied to improve training issues, such as the above Figure 6 And the following about Fig.13 and 14 , but reasoning-time techniques may also be implemented. In various embodiments, in such cases, partial word-gram suggestion errors may be corrected using constrained prefix matching for generating next word-gram predictions. Fig.11 is a high-level flow chart illustrating techniques and methods for implementing constrained prefix matching for generating next word-gram predictions in accordance with some embodiments.

[0125] In some embodiments, as indicated at 1110, input text may be received to perform next word prediction on the input text. For example, the input text may be received as part of a request for code suggestions, as described above with respect to Figure 4 , 8 10 and may be the amount of code written in the programming language before the cursor when the code suggestion request is triggered. Input text for other text prediction or completion scenarios may also be received (e.g., text auto-completion for various use cases, such as text suggestions for drafting various documents, providing input for other requests or natural language forms, etc.).

[0126] As indicated at 1120, the word boundaries of the input text relative to the word segmenter can be determined. The rightmost boundary may contain a partial word unit. This partial word unit can be used as a prompt suffix to constrain the next word unit generation. Note: These word boundaries are larger word unit units and are generally referred to as pre-word units. The last word unit can be a partial word unit. For example, a variety of different word segmenters or word unitization techniques can be used. Word units can be individual words in a sentence, including the pre-space character "[space] word" before the word as a word unit. Punctuation, empty spaces, carriage returns, or various other characters can also be grouped into or individually considered as word units.

[0127] In some embodiments, a "previous word" that occurs immediately before a partial word can be identified. From these previous words, backtracking words can be determined. For example, starting from the previous word that immediately precedes the prompt suffix, one or more of the previous words can be added to the backtracking word, and the backtracking operation is performed in the order of the words until the maximum number of backtracking words is reached or a special character (e.g., a carriage return) is reached. In some embodiments, the maximum number of backtracking words can be a configurable parameter for next word prediction (e.g., as part of a request for next word prediction or as a separate configuration request).

[0128] In various embodiments, one or more possible word-grams that match the hint suffix may be identified from the possible word-grams as indicated at 1130. For example, the possible word-grams may be a vocabulary of a language (e.g., a programming or human language) that may have different predictable words. A match may be identified when the hint suffix matches at the beginning or end of a possible word-gram (e.g., a possible word-gram matches when the hint suffix [SYS] matches the beginning of a possible word-gram [SYS* or the end of a possible word-gram *SYS]).

[0129] In some embodiments, one or more different data structures can be used to identify matching possible word units. For example, a trie data structure can be used to store different possible word units. A trie can be a search tree of prefixes (or suffixes), where the trie can be a string index for a vocabulary of words, where individual nodes contain links to suffix child nodes, where the links add additional characters to the suffix at each child node. Another example of a data structure that can be used to efficiently identify matching word units can be a cache of possible matching word units (e.g., as a Boolean mask).

[0130] In some embodiments, as indicated at 1140, the next word-gram prediction can be filtered based on the one or more possible word-grams identified, wherein the next word-gram prediction is generated by applying the machine learning model to the remainder of the input text, the remainder of the input text not including multiple backtracked word-grams corresponding to the previous word-gram. For example, a given input to the machine learning model may have multiple input word-grams (including partial word-grams) from the text before the cursor, such as 15 word-grams, where word-gram 15 is a partial word-gram. If the number of backtracked word-grams is 3, then word-grams 14, 13, and 12 (neighboring word-gram 15) may not be used as input to the next word-gram prediction machine learning model, so that the input may instead be the previous word-grams from word-gram 11 backward to 1 (and may include an additional 3 previous word-grams to supplement the backtracked word-grams) and partial word-gram 15.

[0131] Given the input word-gram, the result of the next word-gram prediction may include multiple different word-gram predictions with different confidence values. Word-gram predictions that are not one of the possible word-grams identified may be disregarded. A remaining highest confidence score of the predictions may be identified.

[0132] For this remaining next word-gram prediction, a number of characters that match the partial word-gram may be subtracted from the left side of the partial word-gram, as indicated at 1150. If no additional characters remain, as indicated at 1160, then the next word-gram prediction may be provided as the next word-gram prediction, as indicated at 1170. If not, then the remaining characters may be used to perform another iteration of next word-gram prediction, as indicated by an active exit from 1160, until no characters remain.

[0133] Fig.12 is a logic block diagram illustrating different iterations of constrained prefix matching according to some embodiments, as described above with respect to Fig.11 Discussion. Iteration 1201 shows the input, which includes the remainder but does not include the backtracking word 1202 and the partial word (e.g., "[space]SYSTE"). The highest confidence, filtered prediction may be "[space]SY". As shown at iteration 1203, this may be used to subtract the matching characters from the partial word, thereby retaining "STE". For this iteration, the prediction may be "[space]SYST". After subtraction, the partial word is "E". Then for iteration 1205, the prediction may be "[space]SYSTEM", which will match the remaining character "E", thereby ending the iteration and using the prediction "[space]SYST EM" as the next word prediction.

[0134] As mentioned above about Figure 1 and 6 As discussed, training data for code suggestion and other text generation systems may encounter difficulties when partial tokens are included as part of the input to text generation reasoning, such as may occur in various auto-completion scenarios where text (e.g., code) is generated to automatically complete the next part of the text given the input context of previously entered text up to a partial token (e.g., a partial or incomplete word). To improve the ability of machine learning models to consider these scenarios, training datasets may have to include data items trained on these partial token scenarios. Fig.13 is a high-level flow chart illustrating techniques and methods for implementing random word-gram partitioning for training a next word-gram prediction model in accordance with some embodiments.

[0135] In some embodiments, as indicated at 1310, text data for training a machine learning model to predict the next text token given an input text token can be received. For example, a request to generate partial token-optimized training data can be received, which can specify a storage location or other information describing a source training data set including text data. In some embodiments, this request can be received or specified as submitted to a machine learning system or service (e.g., implemented as a method similar to Figure 2 Part of a training job of a machine learning service that is part of a provider network 200 in the provider network.

[0136] According to some embodiments, as indicated at 1320, a plurality of tokens may be determined from the text data. For example, a variety of different tokenizers may be applied to generate tokens from the input text. For example, a token may be an individual word in a sentence, including a space character "[space] word" before the word as a token. Punctuation marks, empty spaces, carriage returns, or various other characters may also be grouped as or individually considered tokens.

[0137] In some embodiments, as indicated at 1330, different word-grams in the plurality of word-grams may be randomly divided into respective sub-word-grams. For example, a sub-word regularization technique may be performed to sample or identify different word-grams that are not optimally divided. Fig.14 Examples of random word-gram segmentation are provided. Such techniques may include randomly selecting a word-gram from among the word-grams determined for the text data (e.g., according to a configurable variable or parameter that may be indicated as a percentage value, wherein the percentage value indicates the likelihood that any one word-gram will be selected for random segmentation). If a word-gram is selected, the word-gram, rather than the source word-gram from which it was generated, may be randomly segmented into sub-word-gram components that are considered to be word-grams.

[0138] In some embodiments, as indicated at 1340, a machine learning model can be trained using a plurality of tokens including corresponding sub-tokens as a training data set to predict the next text token given an input text token. In some embodiments, the trained machine learning model can be stored at a specified location in the training request. In some embodiments, the machine learning model can be deployed for different applications, including code suggestion or text auto-completion applications as discussed above. A variety of different training techniques and machine learning model types can be used for next token prediction, such as sequence-to-sequence models or other neural network-based models, such as long short-term memory (LTSM).

[0139] Fig.14 1410, text data, such as "New York", may be an instance of a longer text string that may be tokenized for a training data set, as described above with respect to Fig.13 . Each word "New" and "York" can be a separate word-gram. Because word-gramming can be performed randomly, such as by using subword regularization techniques, different instances can be generated, as indicated at 1420, 1430, and 1440 (where each box is a sub-word-gram). In these instances, a random selection is made as to whether a word-gram is selected for further partitioning (e.g., both in 1420, "York" in 1430, and "New" in 1440). In some embodiments, the random partitioning may produce two or more fragments, and may include treating individual characters as sub-word-grams.

[0140] As mentioned above about Figure 1 and 7 As discussed, it may be difficult to obtain high-quality data sets for training and evaluating different systems, services or applications, such as those data sets implemented to provide code or other text suggestions as discussed above. Some data sets, such as evaluation data sets, may be highly specialized. For example, the evaluation data set of the code suggestion system may rely on various code hints or problems to be solved. Code hints can describe programming problems that can be obtained by the code suggestion system, and then generate corresponding code parts to solve or meet the problems. For example, a hint can be considered as a function for checking whether a given number is odd or even. Various example claims or unit tests that can be used to determine whether the generated code returns the correct value that satisfies the test may be included. In order to increase the number of available high-quality evaluation data sets, the technology of generating new high-quality evaluation data sets may be highly desirable.

[0141] Fig.15 is a high-level flow chart illustrating techniques and methods for implementing programmatically generating evaluation data sets for a code generation system according to some embodiments. These techniques can be implemented by various types of systems or other code generation systems for testing, developing, or implementing code suggestions. In some embodiments, a programming language conversion system can include a set of tools, including tools for programmatically generating evaluation data sets for a code generation system.

[0142] In some embodiments, as indicated at 1510, an evaluation data set specified in a first programming language may be received, wherein different items of the evaluation data set correspond to different corresponding evaluation tests of the code generation system. For example, accessing and obtaining one or more files, objects, locations, or other information of the evaluation data set may be provided as part of a request to perform a conversion of the evaluation data set from a first (e.g., source) programming language to a second (e.g., target) programming language. In some embodiments, a plurality of target programming languages ​​may be specified as part of the request, and therefore specified as a plurality of performances of the technology, as discussed below.

[0143] In some embodiments, as indicated at 1520, individual items in different items of the data set can be converted to a second programming language. For example, conversion of the prompt can include changing features, such as symbols used to indicate code comments (e.g., non-executable statements in the code), such as changing from """"" to " / **". Conversion of the prompt can also include changing natural language statements to replace source programming language terms with target programming language terms, changing "write a function in Python" to "write a function in Java", to... or changes between terms, such as "none" to "empty". In some scenarios, the conversion rules can remove unnecessary or non-convertible source programming language specific statements.

[0144] As indicated at 1530, the function signature of the item in the first programming language can be converted to the second programming language. For example, the function signature can be identified by parsing the items of the evaluation data set according to a parser or rule set of the first (source) programming language to locate the function signature. For example, in the python programming language, a search (e.g., a regular expression search) for "def" can be performed to locate the function signature, which can also be defined using various other symbols (e.g., can include independent variables within parentheses).

[0145] Once the function signature is located, different techniques can be performed to determine the types of the corresponding arguments or parameters of the function. For example, the test case for the function can identify the values ​​of the arguments. For example, in Fig.16A , the function signature 1612 may provide the transformed data set source item 1610. The function signature 1612 may have been located (as discussed above) and then evaluated to determine the value of the independent variable "cost, m, n". One of the test cases indicates that the candidate function has inputs such as "[[1,2,3],[4,8,2],[1,5,3]],2,2", which can be decomposed into a cost of "[[1,2,3],[4,8,2],[1,5,3]]", m is "2" and n is "2". Therefore, cost may be a list, and m and n may be int's. The return value may also be inferred from the test, for example "[[1,2,3],[4,8,2],[1,5,3]],2,2==8", where "8" is the expected return value of the test case. In some embodiments, the return value determination may include executing both the source test item function and the test to identify the returned value.

[0146] To accomplish the conversion, one or more mapping rules specific to the conversion of function signatures from the source programming language to the target programming language may be applied. Fig.16A and 16B In the example shown, the mapping rule is applied to convert the text of the Python function signature 1612 into Java 1624, where "def" is converted into "class MinCost{public static in MinCost(List <List <integer>>cost, int m, int n{". As discussed above, the determination of the independent variables can allow the determined source value types to be mapped to the target value types explicitly declared in the Java instance 1624 (e.g., "List", "integer", and "integer"). In some scenarios, the independent variables are lists of heterogeneous data (which can be lists, sets, etc., as elements). To perform the conversion, the reasoning technique can check the data type by recursively looking at the elements. In such techniques, the most general type given all the values ​​can be selected. For example, if an independent variable value of type is List <integer>And another independent variable value of type List <double>, the type should be List <double>In another example, if an argument type is List <List <integer>> and the other argument type is List <HashMap<Integer,String> >, then the type will be List <object>.

[0147] In some embodiments, as indicated at 1540, the test statements of the project in the first programming language can be converted to the second programming language. Some knowledge determined as part of the conversion function signature can be used to convert the test statements. For example, the variable format "(cost, m, n)" can be used to extract various test values ​​from 1618 and insert them into the target programming language version of the test, such as converting from "assert candidate" to "class Main{public static voidmain(String[]args)throws Exception{if(!(MinCost.MinCost(Arrays.asList(Arrays.asList(1, 2, 3),Arrays.asList(4,8,2),Arrays.asList(1,5,3)),2,2)==8)throw new java.lang.Exception("Exception--test case 0 did not pass");}", as indicated in test statement 1628. This can be repeated in each test.

[0148] In some embodiments, as indicated at 1550, the body of the converted function signature can be generated in a second programming language based on a hint in a project used as input to a machine learning model that is trained to generate code in the second programming language. Figure 1 , 2 , 4 and 10, the code generation technique can take as input a given portion of code or a natural language statement describing the code to be generated, and generate a suggested code. Fig.15 Other systems, services, or applications of the technology can call the code generation system or the locally implemented machine learning model through an interface to generate function body code in a target programming language (for example, by specifying the target programming language so that a machine learning model specific to the target programming language is used).

[0149] In various embodiments, the assembly of different converted project portions may be performed according to one or more conversion rules of the project of the source programming to the evaluation data set of the target programming language. For example, the ordering of the portions in one programming language may be changed. For example, in Fig.16A , the ordering of the test items is function signature 1612, then hint 1614, then function body 1616, and finally test statements 1618, where as in the converted version of 1620, the ordering is hint 1622, then function signature 1624, then function body 1626, and finally test statements 1628. The ordering required for different programming languages ​​may vary, which can be taken into account when assembling the converted version.

[0150] In some embodiments, as indicated at 1560, the converted individual evaluation data sets of different evaluation data sets in the evaluation data set can be stored as part of a new evaluation data set. For example, each item of the evaluation data set can be a different file, document or other object. When each new converted item is created, the corresponding different files, documents or objects can be added to the target storage location of the new evaluation data set. In some embodiments, various errors can trigger notifications and / or store source items in separate storage locations for manual conversion (e.g., sending a notification that the source items should be reviewed).

[0151] In various embodiments, the methods described herein may be implemented by any combination of hardware and software. For example, in one embodiment, the methods may be implemented by a computer system including one or more processors (e.g., Fig.17 The method of the present invention is implemented in a computer system (in a computer system) in which one or more processors execute program instructions stored on a computer-readable storage medium coupled to the processor. The program instructions can be configured to implement the functions described herein (e.g., to implement the functions of various servers and other components of the provider network described herein). The various methods shown in the accompanying drawings and described herein represent example embodiments of the method. The order of any method can be changed, and various elements can be added, reordered, combined, omitted, modified, etc.

[0152] The techniques discussed above may be executed on one or more computer systems, which may interact with various other devices. Fig.17 2000 is a block diagram illustrating an example computer system according to various embodiments. For example, in various embodiments, computer system 2000 may be configured to implement a node of a computing cluster, a distributed key-value data store, and / or a client. Computer system 2000 may be any of various types of devices, including but not limited to a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a portable computer, a workstation, a network computer, a consumer device, an application server, a storage device, a telephone, a mobile phone, or generally any type of computing device.

[0153] The computer system 2000 includes one or more processors 2010 (any of which may include multiple cores, which may be single-threaded or multi-threaded) coupled to a system memory 2020 via an input / output (I / O) interface 2030. The computer system 2000 further includes a network interface 2040 coupled to the I / O interface 2030. In various embodiments, the computer system 2000 may be a single processor system including one processor 2010, or a multiprocessor system including several processors 2010 (e.g., two, four, eight, or another suitable number). The processor 2010 may be any suitable processor capable of executing instructions. For example, in various embodiments, the processor 2010 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, PowerPC, SPARC, or MIPS ISAs, or any other suitable ISAs. In a multiprocessor system, each processor in the processor 2010 may typically, but not necessarily, implement the same ISA. The computer system 2000 also includes one or more network communication devices (e.g., network interface 2040) for communicating with other systems and / or components over a communication network (e.g., the Internet, a LAN, etc.). For example, a client application executing on the system 2000 can use the network interface 2040 to communicate with a server application executing on a single server or server cluster that implements one or more of the components of the provider network described herein. In another example, an instance of a server application executing on the computer system 2000 can use the network interface 2040 to communicate with other instances of the server application (or another server application) that can be implemented on other computer systems (e.g., computer system 2090).

[0154] In the illustrated embodiment, the computer system 2000 also includes one or more persistent storage devices 2060 and / or one or more I / O devices 2080. In various embodiments, the persistent storage device 2060 may correspond to a disk drive, a tape drive, a solid-state memory, other mass storage devices, or any other persistent storage device. The computer system 2000 (or a distributed application or operating system running thereon) may store instructions and / or data in the persistent storage device 2060 as desired, and may retrieve the stored instructions and / or data as needed. For example, in some embodiments, the computer system 2000 may host a storage system server node, and the persistent storage device 2060 may include an SSD attached to the server node.

[0155] The computer system 2000 includes one or more system memories 2020 configured to store instructions and data accessible by the processor 2010. In various embodiments, the system memory 2020 can be implemented using any suitable memory technology, (e.g., one or more of the following: cache, static random access memory (SRAM), DRAM, RDRAM, EDO RAM, DDR 20RAM, synchronous dynamic RAM (SDRAM), Rambus RAM, EEPROM, non-volatile / flash memory, or any other type of memory). The system memory 2020 can contain program instructions 2025 that can be executed by the processor 2010 to implement the methods and techniques described herein. In various embodiments, the program instructions 2025 can be encoded in platform native binary, any interpreted language such as JavaTM byte code, or any other language such as C / C++, JavaTM, or any combination thereof. For example, in the illustrated embodiment, the program instructions 2025 include program instructions that can be executed to implement the functions of the provider network in different embodiments. In some embodiments, the program instructions 2025 can implement multiple separate clients, server nodes, and / or other components.

[0156] In some embodiments, program instructions 2025 may include instructions executable to implement an operating system (not shown), which may be any of various operating systems such as UNIX, LINUX, SolarisTM, MacOSTM, WindowsTM, etc. Any or all program instructions 2025 may be provided as a computer program product or software, which may include a non-transitory computer-readable storage medium having instructions stored thereon, which may be used to program a computer system (or other electronic device) to perform processes according to various embodiments, such as various techniques for finding matching code sources based on indexing and comparing similarities. Non-transitory computer-readable storage media may include any mechanism for storing information in a machine (e.g., computer) readable form (e.g., software, processing application). In general, non-transitory computer-accessible media may include computer-readable storage media or memory media, such as magnetic media or optical media, for example, a disk or DVD / CD-ROM coupled to the computer system 2000 via an I / O interface 2030. The non-transitory computer-readable storage medium may also include any volatile or non-volatile medium (such as RAM (e.g., SDRAM, DDR SDRAM, RDRAM, SRAM, etc.), ROM, etc.), which may be included in some embodiments of the computer system 2000 as system memory 2020 or another type of memory. In other embodiments, program instructions may be transmitted using optical, acoustic or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.) through a communication medium such as a network and / or wireless link, such as may be implemented via the network interface 2040.

[0157] In some embodiments, system memory 2020 may include data storage 2045, which may be configured as described herein. In general, system memory 2020 (e.g., data storage 2045 within system memory 2020), persistent storage 2060, and / or remote storage 2070 may store data blocks, copies of data blocks, metadata associated with data blocks and / or their states, configuration information, and / or any other information useful for implementing the methods and techniques described herein.

[0158] In one embodiment, the I / O interface 2030 may be configured to coordinate I / O traffic between the processor 2010, the system memory 2020, and any peripheral devices in the system, including through the network interface 2040 or other peripheral interfaces. In some embodiments, the I / O interface 2030 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 2020) into a format suitable for use by another component (e.g., the processor 2010). In some embodiments, for example, the I / O interface 2030 may include support for devices attached through various types of peripheral buses (such as variants of the peripheral component interconnect (PCI) bus standard or the universal serial bus (USB) standard). In some embodiments, for example, the functionality of the I / O interface 2030 may be split into two or more separate components, such as a north bridge and a south bridge. Moreover, in some embodiments, some or all of the functionality of the I / O interface 2030 (such as the interface of the system memory 2020) may be directly incorporated into the processor 2010.

[0159] For example, the network interface 2040 can be configured to allow data to be exchanged between the computer system 2000 and other devices attached to the network, such as other computer systems 2090 (which can implement one or more storage system server nodes, database engine head nodes, and / or clients of the database system described herein). In addition, the network interface 2040 can be configured to allow communication between the computer system 2000 and various I / O devices 2050 and / or remote storage devices 2070. In some embodiments, the input / output devices 2050 may include one or more display terminals, keyboards, keypads, touch pads, scanning devices, voice or optical recognition devices, or any other devices suitable for inputting or retrieving data by one or more computer systems 2000. Multiple input / output devices 2050 may exist in the computer system 2000 or may be distributed on different nodes of a distributed system including the computer system 2000. In some embodiments, similar input / output devices may be separate from the computer system 2000 and may interact with one or more nodes of a distributed system including the computer system 2000 via a wired or wireless connection (such as through the network interface 2040). The network interface 2040 may generally support one or more wireless networking protocols (e.g., Wi-Fi / IEEE 802.11 or another wireless networking standard). However, in various embodiments, the network interface 2040 may support communications over any suitable wired or wireless general purpose data network (e.g., other types of Ethernet). Additionally, the network interface 2040 may support communications over a telecommunications / telephone network (e.g., an analog voice network or a digital fiber optic communications network), over a storage area network (e.g., a Fibre Channel SAN), or over any other suitable type of network and / or protocol. In various embodiments, the computer system 2000 may include, for example, a wireless network such as a Wi-Fi LAN or a wireless LAN. Fig.17 More, fewer, or different components may be shown (eg, a display, a video card, a sound card, peripheral devices, other network interfaces such as an ATM interface, an Ethernet interface, a frame relay interface, etc.).

[0160] It should be noted that any distributed system embodiment or any component thereof in the distributed system embodiment described herein can be implemented as one or more network-based services. For example, the computing cluster in the computing service can present computing services to the client and / or adopt other types of services of the distributed computing system described herein as network-based services. In some embodiments, network-based services can be implemented by software and / or hardware systems designed to support interoperable machines on the network to machine interactions. Network-based services can have interfaces described in machine-processable formats (e.g., Web Services Description Language (WSDL)). Other systems can interact with network-based services in a manner specified by the description of the interface of network-based services. For example, network-based services can define various operations that other systems can call, and can define specific application programming interfaces (APIs) that other systems can follow when requesting various operations.

[0161] In various embodiments, a network-based service may be requested or invoked using a message that includes parameters and / or data associated with the network-based service request. Such messages may be formatted according to a particular markup language such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol such as Simple Object Access Protocol (SOAP). To execute a network-based service request, a network-based service client may assemble a message including the request using an Internet-based application layer transport protocol such as Hypertext Transfer Protocol (HTTP), and transmit the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the network-based service.

[0162] In some embodiments, a network-based service may be implemented using a representational state transfer ("RESTful") technique rather than a message-based technique. For example, a network-based service implemented according to RESTful techniques may be invoked via parameters included in an HTTP method (such as PUT, GET, or DELETE) rather than encapsulated in a SOAP message.

[0163] Although the above embodiments have been described in considerable detail, numerous changes and modifications may be made, as will become apparent to those skilled in the art once the above disclosure is fully understood. The following claims are intended to be interpreted as covering all such modifications and changes, and the above description should therefore be regarded as illustrative rather than restrictive.

[0164] Embodiments of the present disclosure may be described in light of the following terms:

[0165] Clause 1. A system comprising:

[0166] at least one processor; and

[0167] a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement an integrated development environment, the integrated development environment being configured to:

[0168] detecting an event that triggers a request for code completion suggestions to include in a code file being edited using a code development system;

[0169] sending a request for code completion suggestions to a code generation system, wherein the request includes one or more characters immediately preceding a cursor at a first time an event that triggers the request for the code completion suggestions is detected;

[0170] receiving the code completion suggestion from the code generation system, the code completion suggestion comprising one or more suggested characters entered into the code file immediately after the cursor at a first time;

[0171] comparing the number of the one or more suggested characters to a corresponding number of one or more actual characters entered into the code file after the first time to determine that the comparison of the number of the one or more suggested characters to the corresponding number of the one or more actual characters satisfies one or more validation criteria; and

[0172] In response to determining that the one or more validation criteria are satisfied, the code completion suggestions are displayed for input into the code file.

[0173] Clause 2. A system according to clause 1, wherein the event that triggers the request is detected based at least in part on a number of keystrokes or an amount of time that has passed since a previous code suggestion or a combination of the number of keystrokes and the amount of time that has passed since a previous code suggestion.

[0174] Clause 3. A system according to any one of clauses 1 to 2, wherein the integrated development environment is further configured to obtain one or more portions of the code file outside of multiple previous tokens as additional context, wherein the code completion suggestions are further based on the obtained additional context of one or more portions of the code file outside of multiple previous tokens.

[0175] Clause 4. The system of any of clauses 1 to 3, wherein the code generation system is implemented as part of a code development service provided by a provider network.

[0176] Clause 5. A method comprising:

[0177] detecting, by an integrated development environment, an event that triggers obtaining code completion suggestions for inclusion in a code file being edited using the integrated development environment;

[0178] causing, by the integrated development environment, when a first time the event triggering a request for code completion suggestions is detected, to generate the code completion suggestions based at least in part on one or more characters immediately preceding a cursor, wherein the code completion suggestions include one or more suggested characters for input into the code file immediately following the cursor at the first time;

[0179] comparing, by the integrated development environment, the number of the one or more suggested characters with a corresponding number of one or more actual characters entered into the code file after the first time to determine that the comparison of the number of the one or more suggested characters with the corresponding number of the one or more actual characters satisfies one or more validation criteria; and

[0180] In response to determining that the one or more validation criteria are satisfied, the code completion suggestions are displayed, by the integrated development environment, for input into the code file.

[0181] Clause 6. A method according to clause 5, wherein the event that triggers the request is detected based at least in part on a number of keystrokes or an amount of time that has passed since a previous code suggestion or a combination of the number of keystrokes and the amount of time that has passed since a previous code suggestion.

[0182] Clause 7. The method of any one of clauses 5 to 6, wherein the event triggering the request is detected based at least in part on the input of a particular character.

[0183] Clause 8. A method according to any one of clauses 5 to 7, further comprising discarding the different code suggestions based on a determination that the one or more validation criteria are not satisfied based on a comparison between additional characters input into the code file and the different code suggestions generated for inclusion in the code file.

[0184] Clause 9. A method according to any one of clauses 5 to 8, further comprising obtaining one or more portions of the code file outside of a plurality of previous tokens as additional context, wherein the code completion suggestion is further based on the obtained additional context of the one or more portions of the code file outside of a plurality of previous tokens.

[0185] Clause 10. The method of Clause 9, wherein obtaining the one or more portions of the code file outside of a previous number of tokens comprises performing one or more regular expression searches.

[0186] Clause 11. The method of clause 5, further comprising obtaining one or more portions of a different code file as additional context, wherein the code completion suggestion is further based on the obtained additional context of the one or more portions of the different code file.

[0187] Clause 12. The method of any one of clauses 5 to 11, wherein causing the code completion suggestion to be generated comprises sending a request for the code completion suggestion to a code development service provided by a provider network.

[0188] Clause 13. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

[0189] detecting an event that triggers obtaining code completion suggestions for inclusion in a code file being edited using a code development system;

[0190] upon detecting a first time the event triggering a request for code completion suggestions, obtaining the code completion suggestions based at least in part on one or more characters immediately preceding a cursor, wherein the code completion suggestions include one or more suggested characters for input into the code file immediately following the cursor at the first time;

[0191] comparing the number of the one or more suggested characters to a corresponding number of one or more actual characters entered into the code file after the first time to determine that the comparison of the number of the one or more suggested characters to the corresponding number of the one or more actual characters satisfies one or more validation criteria; and

[0192] In response to determining that the one or more validation criteria are satisfied, the code completion suggestions are displayed for input into the code file.

[0193] Clause 14. One or more non-transitory computer-readable storage media as described in Clause 13, wherein the event that triggers the request is detected based at least in part on a number of keystrokes or an amount of time that has elapsed since a previous code suggestion or a combination of the number of keystrokes and the amount of time that has elapsed since a previous code suggestion.

[0194] Clause 15. The one or more non-transitory computer-readable storage media of any one of Clauses 13 to 14, wherein the event triggering the request is detected based at least in part on the entry of a particular character.

[0195] Clause 16. One or more non-transitory computer-readable storage media according to any one of clauses 13 to 15, storing additional program instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement discarding the different code suggestions based on a determination that the one or more validation criteria are not satisfied based on a comparison between additional characters input into the code file and the different code suggestions generated for inclusion in the code file.

[0196] Clause 17. One or more non-transitory computer-readable storage media according to any one of clauses 13 to 16, storing additional program instructions, which, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement obtaining one or more portions of the code file outside of multiple previous tokens as additional context, wherein the code completion suggestions are further based on the obtained additional context of one or more portions of the code file outside of multiple previous tokens.

[0197] Clause 18. One or more non-transitory computer-readable storage media according to Clause 17, wherein, upon obtaining the one or more portions of the code file outside of the previous plurality of tokens, the program instructions cause the one or more computing devices to perform one or more regular expression searches.

[0198] Clause 19. One or more non-transitory computer-readable storage media according to clause 13, storing additional program instructions, which, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement obtaining one or more portions of different code files as additional context, wherein the code completion suggestions are further based on the additional context of the obtained one or more portions of the different code files.

[0199] Clause 20. One or more non-transitory computer-readable storage media according to any one of clauses 13 to 19, wherein, when obtaining the code completion suggestion, the program instructions cause the one or more computing devices to implement sending a request for the code completion suggestion to a code development service provided by a provider network.

[0200] Clause 21. A system comprising:

[0201] at least one processor; and

[0202] a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a programming language conversion system, the programming language conversion system being configured to:

[0203] receiving, through an interface of the programming language conversion system, a request to convert an evaluation data set specified in a first programming language, wherein different items of the evaluation data set correspond to different respective evaluation tests of a code generation system;

[0204] converting individual ones of the different items of the data set into a second programming language;

[0205] converting a function signature of the item in the first programming language into the second programming language;

[0206] converting one or more test statements of the project in the first programming language into the second programming language;

[0207] sending a request to the code generation system to generate a body of the converted function signature in the second programming language according to the prompt in the project; and

[0208] receiving the body of the function signature in the second programming language from the code generation system; and

[0209] The transformed individual ones of the different items of the evaluation data set are stored as part of a new evaluation data set.

[0210] Clause 22. The system of clause 21, wherein the second programming language is specified in the request to convert the evaluation data set.

[0211] Clause 23. A system according to any one of clauses 21 to 22, wherein in order to convert the function signature of the project in the first programming language into the second programming language, the programming language conversion system is further configured to identify the corresponding type of the first programming language for one or more parameters in the function signature, and the one or more parameters are mapped to the corresponding type of the second programming language.

[0212] Clause 24. A system according to any one of Clauses 21 to 23, wherein in order to convert the one or more test statements of the project in the first programming language into the second programming language, the programming language conversion system is further configured to identify the corresponding types of the first programming language for one or more parameters in the one or more test statements, and the one or more parameters are mapped to the corresponding types of the second programming language.

[0213] Clause 25. A method comprising:

[0214] receiving, through an interface of the programming language conversion system, an evaluation data set specified in a first programming language, wherein different items of the evaluation data set correspond to different respective evaluation tests of a code generation system;

[0215] converting individual ones of the different items of the data set into a second programming language;

[0216] converting, by the programming language conversion system, a function signature of the project in the first programming language into the second programming language;

[0217] converting, by the programming language conversion system, one or more test statements of the project in the first programming language into the second programming language; and

[0218] causing, by the programming language conversion system, the body of the converted function signature to be generated in the second programming language based on hints in the project used as input to a machine learning model trained to generate code in the second programming language; and

[0219] The transformed individual ones of the different items of the evaluation data set are stored as part of a new evaluation data set.

[0220] Clause 26. The method of Clause 25, further comprising receiving a request to convert the evaluation data set specifying the second programming language.

[0221] Clause 27. The method according to any one of clauses 25 to 26, further comprising:

[0222] converting individual ones of the different items of the data set into a third programming language;

[0223] The programming language conversion system converts the function signature of the project in the first programming language into the third programming language;

[0224] converting, by the programming language conversion system, the one or more test statements of the project in the first programming language into the third programming language; and

[0225] causing, by the programming language conversion system, the body of the converted function signature in the third programming language to be generated in the third programming language based on the hint in the project used as input to a second machine learning model trained to generate code in the third programming language; and

[0226] The converted individual ones of the different items of the evaluation data set are stored in the third programming language as part of a second new evaluation data set.

[0227] Clause 28. A method according to any one of clauses 25 to 27, wherein converting the function signature of the project in the first programming language into the second programming language includes identifying the corresponding types of the first programming language for one or more parameters in the function signature, and the one or more parameters are mapped to the corresponding types of the second programming language.

[0228] Clause 29. A method according to any one of clauses 25 to 28, wherein converting the one or more test statements of the project in the first programming language into the second programming language includes identifying the corresponding types of one or more parameters in the first programming language in the test statement, and the one or more parameters are mapped to the corresponding types of the second programming language.

[0229] Clause 30. The method of any one of Clauses 25 to 29, wherein the programming language conversion system is implemented as part of a code development service provided by a provider network, and wherein a request to perform the conversion is received from a client of the provider network.

[0230] Clause 31. The method of any one of Clauses 25 to 30, further comprising performing a natural language conversion on a portion of the prompt according to the second programming language.

[0231] Clause 32. A method according to any one of clauses 25 to 31, wherein causing the body of the converted function signature to be generated in the second programming language comprises sending a request to a code generation system, the code generation system being implemented as part of a code development service provided by a provider network.

[0232] Clause 33. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

[0233] receiving, through an interface of the programming language conversion system, an evaluation data set specified in a first programming language, wherein different items of the evaluation data set correspond to different respective evaluation tests of a code generation system;

[0234] converting individual ones of the different items of the data set into a second programming language;

[0235] converting, by the programming language conversion system, a function signature of the project in the first programming language into the second programming language;

[0236] converting, by the programming language conversion system, one or more test statements of the project in the first programming language into the second programming language; and

[0237] causing, by the programming language conversion system, the body of the converted function signature to be generated in the second programming language based on hints in the project used as input to a machine learning model trained to generate code in the second programming language; and

[0238] The transformed individual ones of the different items of the evaluation data set are stored as part of a new evaluation data set.

[0239] Clause 34. One or more non-transitory computer-readable storage media according to clause 33, storing additional programming instructions, which, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement receiving a request to convert the evaluation data set specifying the second programming language.

[0240] Clause 35. One or more non-transitory computer-readable storage media according to any of clauses 33 to 34, storing further programming instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement:

[0241] converting individual ones of the different items of the data set into a third programming language;

[0242] The programming language conversion system converts the function signature of the project in the first programming language into the third programming language;

[0243] converting, by the programming language conversion system, the one or more test statements of the project in the first programming language into the third programming language; and

[0244] causing, by the programming language conversion system, the body of the converted function signature in the third programming language to be generated in the third programming language based on the hint in the project used as input to a second machine learning model trained to generate code in the third programming language; and

[0245] The converted individual ones of the different items of the evaluation data set are stored in the third programming language as part of a second new evaluation data set.

[0246] Clause 36. One or more non-transitory computer-readable storage media according to any one of clauses 33 to 36, wherein, when converting the function signature of the project in the first programming language into the second programming language, the program instructions cause the one or more computing devices to implement recognition of the corresponding types of the first programming language for one or more parameters in the function signature, and the one or more parameters are mapped to the corresponding types of the second programming language.

[0247] Clause 37. One or more non-transitory computer-readable storage media according to any one of clauses 33 to 36, wherein, when converting the function signature of the project in the first programming language into the second programming language, the program instructions cause the one or more computing devices to implement identification of the corresponding types of the first programming language for one or more parameters in the test statement, and the one or more parameters are mapped to the corresponding types of the second programming language.

[0248] Clause 38. One or more non-transitory computer-readable storage media according to any one of clauses 33 to 37, wherein the programming language conversion system is implemented as part of a code development service provided by a provider network, and wherein a request to perform the conversion is received from a client of the provider network.

[0249] Clause 39. One or more non-transitory computer-readable storage media according to any one of clauses 33 to 38, storing additional programming instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement natural language conversion of a portion of the prompt according to the second programming language.

[0250] Clause 40. One or more non-transitory computer-readable storage media according to any one of clauses 33 to 39, wherein, when causing the body of the converted function signature to be generated in a second programming language, the program instructions cause the one or more computing devices to implement sending a request to a code generation system, wherein the code generation system is implemented as part of a code development service provided by a provider network.

[0251] Clause 41. A system comprising:

[0252] at least one processor; and

[0253] a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a code generation system, the code generation system being configured to:

[0254] receiving input programming code to perform next word prediction on the input programming code;

[0255] Determining word boundaries of the input text relative to a word segmenter, wherein the rightmost boundary contains a partial word-gram, the partial word-gram being used as a prompt suffix;

[0256] identifying one or more possible word-grams matching the prompt suffix from a plurality of possible word-grams by starting with the prompt suffix or ending with the prompt suffix;

[0257] filtering a next word-gram prediction based on the identified one or more possible word-grams, wherein the next word-gram prediction is generated by applying a machine learning model trained to predict a next word-gram of programming code to a remaining portion of the input programming code that does not include a plurality of backtracking word-grams corresponding to the previous word-gram, wherein filtering is performed for one or more iterations, the filtering removing one or more characters that match one of the next word-gram predictions from the left side of the partial word-gram after each iteration until no characters remain in the partial word-gram; and

[0258] A last word-gram prediction in the next word-gram predictions is provided as a next word-gram prediction for the input programming code.

[0259] Clause 42. A system according to Clause 41, wherein in order to identify the one or more possible word-grams that match the partial word-gram from the multiple possible word-grams, the code generation system is configured to access a dictionary tree data structure storing the multiple possible word-grams.

[0260] Clause 43. A system according to any one of clauses 41 to 42, wherein in order to identify the one or more possible word-grams that match the partial word-gram from the multiple possible word-grams, the code generation system is configured to access a mask cache storing possible partial word-grams.

[0261] Clause 44. A system according to any of clauses 41 to 43, wherein the code generation system is implemented as part of a code development service provided by a provider network, and wherein the input programming code is received as part of a request to generate a code suggestion for a code file received from a client of the provider network.

[0262] Clause 45. A method comprising:

[0263] receiving, by a text generation system, an input text to perform next word-gram prediction on the input text;

[0264] determining, by the text generation system of the input text relative to a word segmenter, word boundaries of the input text, wherein a rightmost boundary contains a partial word-gram, the partial word-gram being used as a hint suffix;

[0265] identifying, by the text generation system, one or more possible word-grams matching the prompt suffix from a plurality of possible word-grams by starting with the prompt suffix or ending with the prompt suffix;

[0266] filtering, by the text generation system, a next word-gram prediction based on the identified one or more possible word-grams, wherein the next word-gram prediction is generated by applying a machine learning model to a remaining portion of the input text that does not include a plurality of backtracked word-grams corresponding to the previous word-gram, wherein filtering is performed for one or more iterations, the filtering removing one or more characters matching one of the next word-gram predictions from the left side of the partial word-gram after each iteration until no characters remain in the partial word-gram; and

[0267] The text generation system provides a last word-gram prediction in the next word-gram prediction as the next word-gram prediction.

[0268] Clause 46. The method of clause 45, wherein identifying the one or more possible word-grams that match the partial word-gram from the plurality of possible word-grams comprises accessing a trie data structure storing the plurality of possible word-grams.

[0269] Clause 47. A method according to any one of clauses 46 to 45, wherein the dictionary tree data structure is used to generate next word prediction using different machine learning models.

[0270] Clause 48. The method of any one of Clauses 45 to 47, wherein identifying the one or more possible word-grams from the plurality of possible word-grams that match the partial word-gram comprises accessing a mask cache storing possible partial word-grams.

[0271] Clause 49. The method according to any one of clauses 45 to 48, further comprising determining, by the text generation system, a number of backtracking word-grams until a maximum number of backtracking word-grams.

[0272] Clause 50. The method of any one of clauses 45 to 49, wherein the input text is received as part of a request to provide the next word-gram prediction, and wherein the next word-gram prediction is provided as a response to the request.

[0273] Clause 51. The method of any one of Clauses 45 to 50, wherein the text generation system is implemented as part of an auto-completion application.

[0274] Clause 52. A method according to any one of clauses 45 to 51, wherein the text generation system is implemented as part of a code development service provided by a provider network, and wherein the input text is received as part of a request to generate code suggestions for a code file received from a client of the provider network.

[0275] Clause 53. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

[0276] receiving input text to perform next word prediction on the input text;

[0277] Determining word boundaries of the input text relative to a word segmenter, wherein the rightmost boundary contains a partial word-gram, the partial word-gram being used as a prompt suffix;

[0278] identifying one or more possible word-grams matching the prompt suffix from a plurality of possible word-grams by starting with the prompt suffix or ending with the prompt suffix;

[0279] filtering a next word-gram prediction based on the identified one or more possible word-grams, wherein the next word-gram prediction is generated by applying a machine learning model to a remaining portion of the input text that does not include a plurality of backtracked word-grams corresponding to the previous word-gram, wherein filtering is performed for one or more iterations, the filtering removing one or more characters matching one of the next word-gram predictions from the left side of the partial word-gram after each iteration until no characters remain in the partial word-gram; and

[0280] A last word-gram prediction in the next word-gram predictions is provided as a next word-gram prediction for the input text.

[0281] Clause 54. One or more non-transitory computer-readable storage media according to Clause 53, wherein, when identifying the one or more possible word-grams that match the partial word-gram from the multiple possible word-grams, the program instructions cause the one or more computing devices to access a dictionary tree data structure storing the multiple possible word-grams.

[0282] Clause 55. One or more non-transitory computer-readable storage media as described in Clause 54, wherein the dictionary tree data structure is used to generate next word-unit predictions using different machine learning models.

[0283] Clause 56. One or more non-transitory computer-readable storage media according to any one of clauses 53 to 55, wherein, when identifying the one or more possible word-grams that match the partial word-gram from the multiple possible word-grams, the program instructions cause the one or more computing devices to implement access to a mask cache storing possible partial word-grams.

[0284] Clause 57. One or more non-transitory computer-readable storage media according to any one of clauses 53 to 56, storing additional program instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement the determination of the number of backtracking words by the text generation system until a maximum number of backtracking words.

[0285] Clause 58. One or more non-transitory computer-readable storage media as described in any of clauses 53 to 57, wherein the input text is received as part of a request to provide the next word prediction, and wherein the next word prediction is provided as a response to the request.

[0286] Clause 59. The one or more non-transitory computer-readable storage media of any one of Clauses 53 to 58, wherein the one or more computing devices are implemented as part of an auto-completion application.

[0287] Clause 60. One or more non-transitory computer-readable storage media as described in any of clauses 53 to 59, wherein the one or more computing devices are implemented as part of a code development service provided by a provider network, and wherein the input text is received as part of a request to generate code suggestions for a code file received from a client of the provider network.

[0288] Clause 61. A system comprising:

[0289] at least one processor; and

[0290] a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a machine learning system, the machine learning system being configured to:

[0291] Receiving programming code for training a machine learning model to predict a next programming code token given an input programming code token;

[0292] parsing the programming code to determine a plurality of word-grams from the programming code;

[0293] randomly dividing different word-grams among the plurality of word-grams into a corresponding plurality of sub-word-grams; and

[0294] Using the plurality of word-grams including the corresponding plurality of sub-word-grams as a training data set, the machine learning model is trained to predict the next programming code word-gram given the programming code word-gram input.

[0295] Clause 62. The system of clause 61, wherein the random partitioning of different word-grams of the word-grams is performed according to a subword regularization technique.

[0296] Clause 63. A system according to any one of clauses 61 to 62, wherein the machine learning model is a pre-trained machine learning model.

[0297] Clause 64. A system according to any of clauses 61 to 63, wherein the machine learning system is implemented as part of a code development service provided by a provider network to train the machine learning model for generating code suggestions.

[0298] Clause 65. A method comprising:

[0299] Receiving, at a machine learning system, text data for training a machine learning model to predict a next text token given an input text token;

[0300] Determining, by the machine learning system, a plurality of word-grams from the text data;

[0301] The machine learning system randomly divides different word-grams among the plurality of word-grams into a corresponding plurality of sub-word-grams; and

[0302] The machine learning model is trained by the machine learning system to predict the next text word-gram given the input text word-gram using the plurality of word-grams including the corresponding plurality of sub-word-grams as a training data set.

[0303] Clause 66. The method according to clause 65, wherein the random partitioning of different word-grams of the word-grams is performed according to a sub-word regularization technique.

[0304] Clause 67. A method according to any one of clauses 64 to 65, wherein the machine learning model is a pre-trained machine learning model.

[0305] Clause 68. A method according to any one of clauses 65 to 67, wherein the text data is a code written in a programming language, and wherein the next text word element and the input text word element are corresponding programming code words.

[0306] Clause 69. A method according to clause 68, wherein determining the multiple word-grams from the text data and randomly dividing different word-grams among the multiple word-grams into corresponding multiple sub-word-grams is performed by a word segmenter, wherein the word segmenter is suitable for training a second machine learning model for predicting the next code word-gram in a second programming language.

[0307] Clause 70. A method according to any one of clauses 65 to 69, further comprising storing the trained machine learning model at a location specified in the request to train the machine learning model.

[0308] Clause 71. A method according to any one of clauses 65 to 70, further comprising causing the trained machine learning model to be deployed as part of an auto-completion application.

[0309] Clause 72. A method according to any one of clauses 65 to 71, wherein the machine learning system is implemented as part of a code development service provided by a provider network to train the machine learning model for generating code suggestions.

[0310] Clause 73. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement:

[0311] Receiving text data for training a machine learning model to predict the next text token given an input text token;

[0312] determining a plurality of word-grams from the text data;

[0313] randomly dividing different word-grams among the plurality of word-grams into a corresponding plurality of sub-word-grams; and

[0314] The machine learning model is trained to predict the next text word-gram given the input text word-gram using the multiple word-grams including the corresponding multiple sub-word-grams as a training data set.

[0315] Clause 74. The one or more non-transitory computer-readable storage media of Clause 73, wherein the random partitioning of different ones of the word-grams is performed according to a subword regularization technique.

[0316] Clause 75. One or more non-transitory computer-readable storage media as described in any of Clauses 73 to 74, wherein the machine learning model is a pre-trained machine learning model.

[0317] Clause 76. One or more non-transitory computer-readable storage media according to any one of clauses 73 to 75, wherein the text data is code written in a programming language, and wherein the next text word and the input text word are corresponding programming code words.

[0318] Clause 77. One or more non-transitory computer-readable storage media according to clause 76, wherein determining the multiple word-grams from the text data and randomly dividing different word-grams among the multiple word-grams into corresponding multiple sub-word-grams are performed by a word segmenter, wherein the word segmenter is suitable for training a second machine learning model for predicting the next code word-gram in a second programming language.

[0319] Clause 78. One or more non-transitory computer-readable storage media according to any one of Clauses 73 to 77, storing additional program instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement storing the trained machine learning model at a location specified in a request to train the machine learning model.

[0320] Clause 79. One or more non-transitory computer-readable storage media as described in any of Clauses 73 to 78, storing additional program instructions that, when executed on or across the one or more computing devices, cause the one or more computing devices to further implement such that the trained machine learning model is deployed as part of an auto-completion application.

[0321] Clause 80. One or more non-transitory computer-readable storage media as described in any of clauses 73 to 79, wherein the machine learning system is implemented as part of a code development service provided by a provider network to train the machine learning model for generating code suggestions.< / object> < / integer> < / double> < / double> < / integer> < / integer> < / t> < / t> < / t> < / t> < / t> < / t> < / t>

Claims

1. A system comprising: at least one processor; and a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement an integrated development environment, the integrated development environment being configured to: detecting an event that triggers obtaining code completion suggestions for inclusion in a code file being edited using the integrated development environment; causing, when a first time the event triggering a request for code completion suggestions is detected, to generate the code completion suggestions based at least in part on one or more characters immediately preceding a cursor, wherein the code completion suggestions include one or more suggested characters for input into the code file immediately following the cursor at the first time; comparing the number of the one or more suggested characters to a corresponding number of one or more actual characters entered into the code file after the first time to determine that the comparison of the number of the one or more suggested characters to the corresponding number of the one or more actual characters satisfies one or more validation criteria; as well as In response to determining that the one or more validation criteria are satisfied, the code completion suggestions are displayed for input into the code file.

2. A system according to claim 1, wherein the event that triggers the request is detected based at least in part on a number of keystrokes or an amount of time that has passed since a previous code suggestion, or a combination of the number of keystrokes and the amount of time that has passed since a previous code suggestion.

3. A system according to any one of claims 1 to 2, wherein the integrated development environment is further configured to obtain one or more parts of the code file outside of multiple previous tokens as additional context, and wherein the code completion suggestion is further based on the obtained additional context of the one or more parts of the code file outside of multiple previous tokens.

4. The system of any one of claims 1 to 3, wherein the code generation system is implemented as part of a code development service provided by a provider network.

5. A method comprising: detecting, by an integrated development environment, an event that triggers obtaining code completion suggestions for inclusion in a code file being edited using the integrated development environment; causing, by the integrated development environment, when a first time the event triggering a request for code completion suggestions is detected, to generate the code completion suggestions based at least in part on one or more characters immediately preceding a cursor, wherein the code completion suggestions include one or more suggested characters for input into the code file immediately following the cursor at the first time; comparing, by the integrated development environment, the number of the one or more suggested characters to a corresponding number of one or more actual characters entered into the code file after the first time to determine that the comparison of the number of the one or more suggested characters to the corresponding number of the one or more actual characters satisfies one or more validation criteria; as well as In response to determining that the one or more validation criteria are satisfied, the code completion suggestions are displayed, by the integrated development environment, for input into the code file.

6. A method according to claim 5, wherein the event that triggers the request is detected based at least in part on a number of key strokes or an amount of time that has passed since a previous code suggestion, or a combination of the number of key strokes and the amount of time that has passed since a previous code suggestion.

7. A method according to any one of claims 5 to 6, wherein the event triggering the request is detected based at least in part on the input of a particular character.

8. The method of any one of claims 5 to 7, further comprising discarding the different code suggestions generated for inclusion in the code file based on a determination that the one or more validation criteria are not satisfied based on a comparison between further characters input into the code file and the different code suggestions generated for inclusion in the code file.

9. The method according to any one of claims 5 to 8, further comprising obtaining one or more parts of the code file outside of multiple previous tokens as additional context, wherein the code completion suggestion is further based on the obtained additional context of the one or more parts of the code file outside of multiple previous tokens.

10. The method of claim 9, wherein obtaining the one or more portions of the code file outside of a previous number of tokens comprises performing one or more regular expression searches.

11. The method of claim 5, further comprising obtaining one or more portions of different code files as additional context, wherein the code completion suggestion is further based on the obtained additional context of the one or more portions of the different code files.

12. The method of any one of claims 5 to 11, wherein causing the code completion suggestion to be generated comprises sending a request for the code completion suggestion to a code development service provided by a provider network.

13. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to implement: detecting an event that triggers obtaining code completion suggestions for inclusion in a code file being edited using a code development system; upon detecting a first time the event triggering a request for code completion suggestions, obtaining the code completion suggestions based at least in part on one or more characters immediately preceding a cursor, wherein the code completion suggestions include one or more suggested characters for input into the code file immediately following the cursor at the first time; comparing the number of the one or more suggested characters to a corresponding number of one or more actual characters entered into the code file after the first time to determine that the comparison of the number of the one or more suggested characters to the corresponding number of the one or more actual characters satisfies one or more validation criteria; as well as In response to determining that the one or more validation criteria are satisfied, the code completion suggestions are displayed for input into the code file.

14. One or more non-transitory computer-readable storage media according to claim 13, wherein the event that triggers the request is detected based at least in part on a number of keystrokes or an amount of time that has passed since a previous code suggestion, or a combination of the number of keystrokes and the amount of time that has passed since a previous code suggestion.

15. One or more non-transitory computer-readable storage media according to any one of claims 13 to 14, wherein: Upon obtaining the code completion suggestion, the program instructions cause the one or more computing devices to implement sending a request for the code completion suggestion to a code development service provided by a provider network.

Images