Enhancing code completion and code synthesis by semantic check

Abstract

A method (600) for providing an auto-fill suggestion in a development environment includes obtaining a user input (120) representing source code generated within the development environment (112) from a user interface (14) executing on a user device (16). The source code is created using a particular programming language and a programming code library (114). The method also includes determining, using a machine learning model (550), at least one auto-fill suggestion (125) based on the user input, the auto-fill suggestion continuing the source code represented by the user input. The method also includes determining, using a rule-based semantic checker (250) configured for the particular programming language, whether the auto-fill suggestion is semantically correct based on the development environment and the programming codelibrary. The method also includes transmitting the auto-fill suggestion for display on the user interface of the user device when the auto-fill suggestion is semantically correct.

Description

Technical Field

[0001] The present disclosure relates to generating autocomplete suggestions for code completion in a development environment. Background Art

[0002] Many computer applications and programs provide auto-fill suggestions for the convenience of the user. Auto-fill suggestions are presented in response to user input and are predictions of one or more characters for continuing or completing the user's input. Typically, auto-fill suggestions are based on previously stored information for a particular user. For example, if a user is typing a name into a form field, the user interface may display a pop-up window with one or more names previously typed by the user in a similar form field or contained in the user's current file or files that the current file references as dependencies. The user may then select a name from the pop-up window to automatically complete the form field without having to type any other additional input.

[0003] Recently, machine learning models have been adapted to generate auto-fill suggestions, which can be used for everything from word processors to programming environments. Machine learning models can be trained with large amounts of data and do not have to be personalized for a specific user. The auto-fill suggestions of a machine learning model can be based on the most common words, phrases, characters, etc. corresponding to the received user input. For example, if a user types "G" in the subject line of an email, the user may be presented with auto-fill suggestions such as "Greetings" or "Good morning." Summary of the invention

[0004] One aspect of the present disclosure provides a computer-implemented method for providing auto-fill suggestions in a development environment. The computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations, including: obtaining user input representing source code generated within the development environment from a user interface executed on a user device. The source code is created using a specific programming language and a programming code library. The operations include using a machine learning model to determine auto-fill suggestions based on the user input, the auto-fill suggestions continuing the source code represented by the user input. The operations also include using a rule-based semantic checker configured for a specific programming language to determine whether the auto-fill suggestions are semantically correct based on the development environment and the programming code library. The operations include transmitting the auto-fill suggestions for display on the user interface of the user device when the auto-fill suggestions are semantically correct.

[0005] Implementations of the present disclosure may include one or more of the following optional features. In some implementations, determining whether the autofill suggestion is semantically correct includes comparing the autofill suggestion to a structural representation of a programming code base. In these implementations, the structural representation of the programming code base can be an abstract tree. In addition, in these implementations, the structural representation of the programming code base can be stored in a memory cache. In addition, the machine learning model and the rule-based semantic checker can be co-located within a cloud computing environment.

[0006] In some implementations, the operations further include determining, using the machine learning model, a plurality of autofill suggestions based on the user input. In these implementations, for each respective autofill suggestion in the plurality of autofill suggestions, the operations include: determining, using a rule-based semantic checker based on the development environment and the programming code base, whether the respective autofill suggestion is semantically correct; and when the respective autofill suggestion is semantically correct, transmitting the respective autofill suggestion for display on a user interface of the user device. In these implementations, the rule-based semantic checker may determine, within the allocated time budget, whether each respective autofill suggestion in the plurality of autofill suggestions is semantically correct. When the rule-based semantic checker is unable to determine whether each respective autofill suggestion in the plurality of autofill suggestions is semantically correct within the allocated time budget, the operation may also include: selecting a portion of the plurality of autofill suggestions to maximize coverage within the allocated time budget; and for each respective autofill suggestion in the portion of the plurality of autofill suggestions, determining whether the respective autofill suggestion is semantically correct based on the development environment and the programming code base using the rule-based semantic checker; and when the respective autofill suggestion is semantically correct, transmitting the respective autofill suggestion for display on a user interface of a user device. In some of these examples, determining whether each respective autofill suggestion in the plurality of autofill suggestions is semantically correct includes: grouping the respective autofill suggestion with similar autofill suggestions from the plurality of autofill suggestions into one or more groups of autofill suggestions; and comparing an example autofill suggestion from each of the one or more groups of autofill suggestions with a structural representation of the programming code base.

[0007] In some implementations, when the autocomplete suggestion is semantically incorrect, the operations include: using a rule-based semantic checker to generate one or more constraints; and using a machine learning model to determine a new autocomplete suggestion based on the user input and the one or more constraints. In some implementations, an integration model facilitates communication between the machine learning model and the rule-based semantic checker.

[0008] Another aspect of the present disclosure provides a system for providing auto-fill suggestions in a development environment. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining user input representing source code generated within a development environment from a user interface executed on a user device. The source code is created using a specific programming language and a programming code library. The operations include using a machine learning model to determine auto-fill suggestions based on the user input, the auto-fill suggestions continuing the source code represented by the user input. The operations also include using a rule-based semantic checker configured for a specific programming language to determine whether the auto-fill suggestions are semantically correct based on the development environment and the programming code library. The operations include transmitting the auto-fill suggestions for display on the user interface of the user device when the auto-fill suggestions are semantically correct.

[0009] This aspect may include one or more of the following optional features. Implementations of the present disclosure may include one or more of the following optional features. In some implementations, determining whether the autofill suggestion is semantically correct includes comparing the autofill suggestion to a structural representation of a programming code base. In these implementations, the structural representation of the programming code base can be an abstract tree. In addition, in these implementations, the structural representation of the programming code base can be stored in a memory cache. In addition, the machine learning model and the rule-based semantic checker can be co-located within a cloud computing environment.

[0010] In some implementations, the operations further include determining, using the machine learning model, a plurality of autofill suggestions based on the user input. In these implementations, for each respective autofill suggestion in the plurality of autofill suggestions, the operations include: determining, using a rule-based semantic checker based on the development environment and the programming code base, whether the respective autofill suggestion is semantically correct; and when the respective autofill suggestion is semantically correct, transmitting the respective autofill suggestion for display on a user interface of the user device. In these implementations, the rule-based semantic checker may determine, within the allocated time budget, whether each respective autofill suggestion in the plurality of autofill suggestions is semantically correct. When the rule-based semantic checker is unable to determine whether each respective autofill suggestion in the plurality of autofill suggestions is semantically correct within the allocated time budget, the operation may also include: selecting a portion of the plurality of autofill suggestions to maximize coverage within the allocated time budget; and for each respective autofill suggestion in the portion of the plurality of autofill suggestions, determining whether the respective autofill suggestion is semantically correct based on the development environment and the programming code base using the rule-based semantic checker; and when the respective autofill suggestion is semantically correct, transmitting the respective autofill suggestion for display on a user interface of a user device. In some of these examples, determining whether each respective autofill suggestion in the plurality of autofill suggestions is semantically correct includes: grouping the respective autofill suggestion with similar autofill suggestions from the plurality of autofill suggestions into one or more groups of autofill suggestions; and comparing an example autofill suggestion from each of the one or more groups of autofill suggestions with a structural representation of the programming code base.

[0011] In some implementations, when the autocomplete suggestion is semantically incorrect, the operations include: using a rule-based semantic checker to generate one or more constraints; and using a machine learning model to determine a new autocomplete suggestion based on the user input and the one or more constraints. In some implementations, an integration model facilitates communication between the machine learning model and the rule-based semantic checker.

[0012] The details of one or more implementations of the present disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a diagram of an example system for generating autocomplete suggestions for code completion in a development environment.

[0014] Figure 2 is a schematic diagram of a rule-based semantic checker communicating with a machine learning model.

[0015] Figure 3 is a diagram of an example abstract syntax tree.

[0016] Figure 4 is a sequence diagram that generates autocomplete suggestions for code completion in the development environment.

[0017] Figure 5 is a diagram of an example training scenario for a machine learning model.

[0018] Figure 6 is a flow diagram of an example arrangement of operations of a method for generating auto-fill suggestions for code completion in a development environment.

[0019] Figure 7 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0020] Like reference numerals in the various drawings indicate like elements. DETAILED DESCRIPTION

[0021] A computer application for code development may provide suggestions for code completion or code synthesis based on received user input. For example, if a user begins typing a function call, the computer application may suggest one or more ways to complete the line of code. Additionally, if a user begins typing code for a loop (such as a "for loop" or a "do while" loop), the computer program may provide one or more suggestions for a script (i.e., multiple lines of code) to complete the loop. These suggestions, referred to as auto-fill suggestions or auto-complete suggestions, may be generated using a machine learning model trained on a large amount of previously received code.

[0022] However, many problems arise when using large-scale machine learning models to generate autocomplete suggestions for code completion. For example, although machine learning models can be regularly fine-tuned or updated to improve accuracy, the suggestions provided are often based on an outdated code base and are therefore less relevant. In addition, code completion suggestions generated using machine learning models are text-based and are not generated with consideration for the content of the code and / or suggestions. In other words, suggestions may be inappropriate because they are semantically incorrect or may have incorrect syntax.

[0023] Implementations herein include a rule-based semantic checker for checking or verifying that autocomplete suggestions are semantically and syntactically correct before presentation. The rule-based semantic checker can be communicatively connected to a user's development environment and code base so that the autocomplete suggestions can be tested against current code associated with the user.

[0024] Figure 1A code suggestion system 100 is shown, which includes a user device 16 having a user interface 14. The user device 16 may correspond to any computing device, such as a desktop workstation, a laptop workstation, or a mobile device (i.e., a smart phone). The user device 16 includes computing resources 17 (e.g., data processing hardware) and / or storage resources 18 (e.g., memory hardware). The user device 16 may be configured to host a development environment 112 and a programming code library 114. The development environment 112, also known as an integrated development environment (IDE), is a software application that facilitates software development and creates / edits / compiles / debugs programming code. The development environment may be specific to a particular language or may be configured to handle multiple programming languages ​​simultaneously. In some implementations, the development environment 112 is executed locally on the user device 106. The programming code library 114 may include a data repository of source code. For example, the code library 114 includes multiple files, libraries, modules, etc., each of which supports or implements one or more programs. In some implementations, the programming code library 114 includes code generated by one or more users 12. As with development environment 112 , programming code library 114 may be stored locally on user device 16 (eg, on memory hardware 18 ), or may be stored remotely on a server or in cloud computing environment 150 .

[0025] In some implementations, the user device 16 communicates with a remote system 150 (also referred to herein as a cloud computing environment) via the network 140. The remote system 150 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) with scalable / elastic resources 152, including computing resources 154 (e.g., data processing hardware) and / or storage resources 156 (e.g., memory hardware). A data repository 158 (i.e., a remote storage device) may be overlaid on the storage resources 146 to allow scalable use of the storage resources 146 by one or more user devices 16 or computing resources 154. The remote system 150 may execute both the machine learning model 550 and the rule-based semantic checker 250 (i.e., the machine learning model 550 and the rule-based semantic checker 250 may be co-located in the cloud computing environment 150). The development environment 112 may be executed locally on the user device 16 (e.g., on the data processing hardware 17) or remotely (e.g., at the remote system 150). Likewise, code library 114 may be stored locally on user device 16 or at a remote system 150 (eg, at data repository 158 ).

[0026] In some implementations, a user 12 of a user device 16 types user input 120 representing source code at the user device 16 via a user interface 14. For example, the user 12 uses a keyboard to type "myFunction(" to start a function call. The user device 16 may then transmit the user input 120 to the machine learning model 550 in real time (i.e., as or shortly after the user provides the user input 120). In response to receiving the user input 120, the machine learning model 550 may be configured to generate one or more auto-fill suggestions 125 for code completion (i.e., continuations or completions of the source code represented by the user input 120) based on the user input 120. In some examples, the one or more suggestions 125 generated by the machine learning model 550 are text-based and may not indicate the actual content of the code (e.g., specific In addition, the machine learning model 550 may not be trained with the latest code in the programming code base 114. For example, one or more users 12 have added code to the code base 114 since the machine learning model 550 was last updated. In order to provide relevant suggestions 125 to the user 12, the one or more suggestions 125 are verified as semantically and / or grammatically correct by the rule-based semantic checker 250 before being presented to the user 12 via the user interface 14 of the user device 16. The rule-based semantic checker 250 can verify each of the one or more suggestions 125 by performing a plurality of predefined checks on each of the one or more suggestions 125. For example, For each of the one or more suggestions 125, the rule-based semantic checker 250 can check resolution (e.g., does the referenced object exist), invocation (e.g., the correct number of actual parameters are passed to the method), assignability (e.g., is the correct type of object passed as a parameter), etc. The rule-based semantic checker 250 can perform multiple predefined checks in a predefined order, with the most important checks (i.e., the checks that most often find errors) performed first. In some implementations, the rule-based semantic checker 250 performs some or all of the multiple checks on each of the one or more suggestions 125 using the structural representation 300 of the programming code base 114. Optionally, when When one or more suggestions 125 are incorrect (e.g., have incorrect grammar), the rule-based semantic checker 250 provides one or more constraints 225 to the machine learning model 550. Then, the machine learning model 550 can generate one or more new suggestions 125 based on the constraints 225 and the user input 120. In some implementations, the machine learning model 550 again provides the newly generated suggestions 125 to the rule-based semantic checker 250 to verify that the suggestions 125 are semantically and / or grammatically correct. In this way, the rule-based semantic checker 250 can iteratively provide constraints 225 to the machine learning model 550 to improve the quality and / or accuracy of the suggestions 125.In other implementations, the cloud environment 150 transmits the new suggestions 125 generated by the machine learning model 550 directly to the user device 16 for display in the user interface 14 without requiring the rule-based semantic checker 250 to verify the newly generated suggestions 125.

[0027] In some implementations, the integrated model 260 facilitates communication between the machine learning model 550 and the rule-based semantic checker 250 (e.g., via an application program interface (API), etc.). As shown, the integrated model 260 can be integrated with the rule-based semantic checker 250. In some implementations, the integrated model 260 can be a standalone application that is also co-located within the cloud computing environment 150.

[0028] The rule-based semantic checker 250 can be programming language specific. That is, the rule-based semantic checker 250 can be configured to check the semantics of the suggestion 125 for a specific programming language. In some implementations, the integrated model 260 is configured to determine or select an appropriate rule-based semantic checker 250 based on the language of one or more suggestions 125. For example, the integrated model 260 can receive one or more suggestions 125 and determine (e.g., via the suggestions 125, user input 120, and / or configuration settings) that the suggestion 125 corresponds to a specific programming language (e.g., C++). The integrated model 260 can then transmit the suggestion 125 to a specific rule-based semantic checker 250 configured for the specific programming language (such as, C++).

[0029] The rule-based semantic checker 250 may perform a number of checks on each suggestion 125 using a structural representation 300 of the programming code base 114. The structural representation 300 may be an abstract syntax tree, as described below with respect to Figure 315. In some implementations, the structure representation 300 is stored in a memory cache 215. By storing the structure representation 300 in a memory cache 215 at or near the semantic checker 250 (e.g., at a remote system 150), the rule-based semantic checker 250 can retrieve the appropriate structure representation 300 more quickly based on the code base 114, user input 120, suggestions 125, etc. Because latency is a major concern in an auto-fill suggestion system (i.e., suggestions must be presented quickly to the user 12 in order to be usable), storing the structure representation 300 in a cache that communicates with the semantic checker 250 allows the semantic checker 250 to minimize latency. In addition, the structure representation 300 can be updated in the memory cache 215 so that the semantic checker 250 determines the appropriate semantics of the suggestion 125 based on the current or latest code in the programming code base 114. When the suggestion 125 is incorrect or inappropriate, the rule-based semantic checker 250 can return one or more constraints 225 to the machine learning model 550 for generating new suggestions 125.

[0030] Figure 1 The system 100 is presented for illustration purposes only and is not intended to be limiting. For example, although only a single example of each component is shown, any number of components 16, 112, 114, 150, 550, and 250 may be communicatively coupled to the system 100. In addition, although some components are shown as being located in the cloud computing environment 150, in some implementations, those components may be hosted locally on the user device 16. Alternatively, although some components are shown as being hosted on the user device 16, in some implementations, those components may be hosted in the cloud computing environment 150. In addition, in various implementations, some or all of the components 112, 114, 550, 250, 260, and 300 are hosted locally on the user device 16, remotely hosted (such as in the cloud computing environment 150), or some combination thereof.

[0031] Figure 2 is an exemplary schematic diagram 200 in which a machine learning model 550 and a rule-based semantic checker 250 are co-located in Figure 1150 in the cloud computing environment 150 of . Here, the machine learning model 550 provides the suggestions 125 to the rule-based semantic checker 250. The rule-based semantic checker 250 then accesses the memory cache 215 to determine whether an appropriate structural representation 300 is available in the cache 215. The appropriate structural representation 300 can be based on the code base 114. When the structural representation is available, the semantic checker 250 retrieves the structural representation 300 of the programming code base 114. As discussed in more detail below, the rule-based semantic checker 250 determines whether one or more suggestions 125 are semantically and / or syntactically correct. In some implementations, when a suggestion 125 is not semantically and / or syntactically correct, the rule-based semantic checker 250 returns a plurality of constraints 225 that the machine learning model 550 uses to generate a new suggestion 125. In other implementations, when at least one suggestion is semantically and / or syntactically correct, the rule-based semantic checker 250 returns a confirmation that the correct suggestion 125 can be displayed to the user. In a further implementation, the rule-based semantic checker 250 returns an indication that one or more suggestions 125 may be displayed to the user in addition to the plurality of constraints 225 for the machine learning model 550 to use in determining new suggestions 125. The integration model 260 may facilitate communication between the machine learning model 550 and the rule-based semantic checker 250.

[0032] Figure 3 is a schematic diagram of an example structural representation 300 of programming code library 114. In this example, structural representation 300 takes the form of an abstract syntax tree. Figure 3 The example of is a simplified version of the abstract syntax tree for illustration purposes. Here, Figure 1 The programming code library 114 includes a function 310 composed of a first sub-function 320 and a second sub-function 330. The first sub-function 320 requires two parameters 321, 322, while the second sub-function 330 requires a single parameter 331. Figure 3, the rule-based semantic checker 250 can quickly perform multiple checks to determine whether the suggestions 125 corresponding to the function 310 are semantically and syntactically correct. For example, the function 310 requires three parameters 321, 322, and 331. Therefore, any suggestion 125 corresponding to the function 310 that has more or less than three parameters in the call will be incorrect (i.e., such a suggestion 125 will fail the method call check). Additionally or alternatively, the structural representation can provide specific information about the components of the code along the branches of the abstract syntax tree. For example, parameter 331 is an integer, while parameter 322 is a text string. In this example, when one or more suggestions 125 call for parameters that are not of the correct type (e.g., Boolean values), the rule-based semantic checker 250 can quickly determine that those suggestions 125 are incorrect (i.e., such a suggestion 125 will fail the assignability check). In another example, the auto-fill suggestion 125 may include "FunctionA(a, b, c)". The rule-based semantic checker 250 may traverse the structural representation 300 to determine whether such a function exists in the programming code base 114. Here, the example structural representation 300 does not include "FunctionA()", and it will be determined that the auto-complete suggestion 125 of "FunctionA(a, b, c)" is incorrect (i.e., such a suggestion 125 will not pass the resolution check).

[0033] In some implementations, the rule-based semantic checker 250 is constrained to determine whether a suggestion 125 is correct within an allocated time budget. Since the suggestions 125 are intended to be displayed to the user in real time, the checks must be performed quickly enough so that the suggestions 125 can be presented to the user while still relevant. Therefore, the rule-based semantic checker 250 can be constrained to quickly and efficiently verify the suggestions 125. For example, if the rule-based semantic checker 250 uses 20 different predefined checks for each suggestion 125 for full verification, the rule-based semantic checker 250 will only perform a subset of the checks that can be performed within the allocated time budget for each suggestion 125, such as the top three checks that maximize coverage of errors found in the suggestions 125. In addition, the rule-based semantic checker 250 can perform checks on one or more suggestions 125 in parallel to save time.

[0034] In another example, when the rule-based semantic checker 250 receives a large number of suggestions 125, it may be inefficient to check each suggestion 125 individually. One way to speed up the checks performed by the rule-based semantic checker 250 is to group similar suggestions 125 and perform checks only on representative suggestions 125 from each group. For example, the rule-based semantic checker 250 receives a plurality of suggestions 125 including "Function(a,b)", "Function(y,z)", "Function(a,b,c)", and "Function(x,y,z)", each representing a call to a function with two or three parameters.

[0035] Here, the rule-based semantic checker 250 can divide the suggestions 125 into two separate groups. The first group can include "Function(a,b)" and "Function(y,z)" because they each include two parameters in the call. The second group can include "Function(a,b,c)" and "Function(x,y,z)" because they each include three parameters in the call. The rule-based semantic checker 250 can then verify or compare one suggestion 125 from each group against the specific code base 114 in which the user 12 is working to determine whether the suggestion is semantically and / or syntactically correct. In the structure representation indicating that the function has three parameters 321, 322, 331, the rule-based semantic checker 250 can verify or compare one suggestion 125 from each group against the specific code base 114 in which the user 12 is working to determine whether the suggestion is semantically and / or syntactically correct. In the structure representation indicating that the function has three parameters 321, 322, 331, the rule-based semantic checker 250 can verify or compare one suggestion 125 from each group against the specific code base 114 in which the user 12 is working Figure 3 In the example shown, the suggestion 125 from the first group is semantically incorrect because the structure representation 300 requires three (3) parameters for the function 310, while the suggestion 125 from the second group has the correct number of parameters in the call.

[0036] In some implementations, when the rule-based semantic checker 250 is unable to check every auto-fill suggestion 125 provided by the model 550 within the allocated time budget, the rule-based semantic checker 250 performs a partial check that maximizes coverage of the auto-fill suggestions 125 given the allocated time budget. For example, the rule-based semantic checker 250 determines groups of auto-fill suggestions 125 (as described above) and then determines which groups provide the broadest coverage of all auto-fill suggestions 125 (e.g., the group with the most auto-fill suggestions 125). The rule-based semantic checker 250 then checks as many groups of auto-fill suggestions 125 as possible within the allocated time budget, starting with the largest group first. In some implementations, the rule-based semantic checker 250 returns only auto-fill suggestions 125 that are confirmed to be correct. In other implementations, the rule-based semantic checker 250 returns all auto-fill suggestions 125, regardless of whether each auto-fill suggestion 125 has been checked.

[0037] The above examples are for illustrative purposes only and are not intended to be limiting. The rule-based semantic checker 250 can be a deterministic model based on a finite set of rules, rather than a machine learning model. Subsequently, the rule-based semantic checker 250 can perform any additional or alternative checks based on the finite set of rules to determine whether the suggestion 125 is correct. The finite set of rules can be based on the expected structural representation 300 of the programming code base 114, or based on a specific programming language. For example, the rule-based semantic checker 250 uses the rule set to verify against the appropriate representation of the programming code base 114 (e.g., the structural representation 300) to determine whether the auto-fill suggestion 125 is semantically and syntactically correct and / or supported in the programming code base 114. In addition, the rule-based semantic checker 250 can also perform any additional or alternative checks based on the rule set to determine whether the code is correct based on the determined specific programming language (e.g., C, C++, Java, Python, etc.).

[0038] Figure 4 An example sequence diagram 400 is shown for providing auto-fill suggestions 125 for code completion in a development environment 112. In some implementations, steps 410 to 460 of the sequence diagram are restricted to be executed within an allocated time budget so that the auto-fill suggestions 125 are provided to the user 12 while the suggestions 125 are still relevant. When steps 410 to 460 cannot be completed in their entirety within the allocated time budget, steps 410 to 460 may be executed for only a portion of the auto-fill suggestions 125.

[0039] The sequence diagram 400 may begin at step 410 by receiving user input 120. The user input 120 may be received via the user interface 14 of the user device 16. In some examples, the user input 120 is received continuously as a streaming input. For example, as the user 12 continues to enter characters while typing, the machine learning model 550 considers each new character when generating auto-fill suggestions 125 because each additional character eliminates the pool of potentially relevant auto-fill suggestions 125. After receiving the user input 120, at step 415, the machine learning model 550 generates one or more auto-fill suggestions 125 based on the user input 120. At step 420, the machine learning model 550 sends / provides the auto-fill suggestions 125 to the rule-based semantic checker 250.

[0040] At step 425, the rule-based semantic checker 250 determines whether each of the one or more auto-fill suggestions 125 is semantically and / or grammatically correct based on the development environment 112 and / or the programming code base 114. For example, the rule-based semantic checker 250 compares the auto-fill suggestion 125 with the structural representation 300 of the programming code base 114. In addition, the rule-based semantic checker 250 can also check whether the auto-fill suggestion 125 is in an appropriate form based on the development environment 112. For example, when the suggestion 125 does not conform to the appropriate grammar, does not include appropriate punctuation, etc., then the suggestion 125 can be determined to be incorrect.

[0041] At step 430, the rule-based semantic checker 250 sends feedback to the machine learning model 550. In some implementations, the rule-based semantic checker 250 sends (as feedback) one or more constraints 225 that limit subsequent suggestions 125 generated or predicted by the machine learning model 550 (i.e., narrow the range of possible suggestions that the machine learning model 550 can predict). Optionally, the rule-based semantic checker 250 sends a list of approved suggestions 125 and a list of rejected suggestions 125 back to the machine learning model 550 (as feedback). If necessary, based on the feedback, the machine learning model 550 generates one or more new suggestions based on the feedback (e.g., constraints 225) provided by the rule-based semantic checker 250 at step 435. In some implementations, after generating a new set of auto-fill suggestions 125, the machine learning model 550 jumps to step 460 and transmits the auto-fill suggestions 125 for display in the user interface 14 executed on the user device 16. In other implementations, the machine learning model 550 first sends the new auto-fill suggestion 125 to the rule-based semantic checker 250 at step 440, rather than jumping to step 460 after generating one or more new suggestions 125 at step 435. Here, at step 445, the rule-based semantic checker 250 checks / validates the new auto-fill suggestion 125 to determine whether the suggestion 125 is semantically correct. At step 450, the rule-based semantic checker 250 again provides feedback, which can take the form of another set of constraints 225 and / or a list of correct suggestions 125 and a list of rejected suggestions 125. In some implementations, when there is at least one correct suggestion 125, the machine learning model transmits the one or more correct suggestions 125 to the user device 16 for display. Although in this example, at step 460, the auto-fill suggestions 125 are transmitted for display at the user device 16, the machine learning model 550 and the rule-based semantic checker 250 can continue to generate suggestions 125 and constraints 225 for any number of cycles. In some implementations, the autofill suggestions 125 are displayed (e.g., via the user interface 14) as a list of selectable options in a drop-down menu. In other implementations, a single autofill suggestion 125 is displayed on the user device 16. For example, the autofill suggestion 125 is displayed as a continuation of the user input 120 (i.e., as a continuation of the sequence of text typed by the user 12), but in a different color, font, size, etc. than the original user input 120. In this example, the autofill suggestion 125 appears to automatically complete the code for the user, and the user can use further input to accept or reject the suggestion 125 (e.g., if the user presses the "enter" key of the keyboard, the autofill suggestion 125 will be accepted, while if the user continues typing, the autofill suggestion 125 will disappear or be replaced by a new autofill suggestion 125).

[0042] Figure 5 An exemplary training process 500 for training a machine learning model 550 is shown. In some implementations, the process 500 employs a two-step training technique. First, the machine learning model 550 is pre-trained with a large set of code to produce a base model. The machine learning model 550 can then be trained in an iterative manner based on additional user input and feedback. For example, the process 500 begins by pre-training the machine learning model 550 using pre-training data 505. Pre-training a model is a technique for initializing a model, which can then be further fine-tuned based on additional training data 510. For the machine learning model 550, pre-training can include starting the machine learning model 550 using pre-training data 505, which includes a large data set that includes code previously written in one or more programming languages.

[0043] The process 500 can then fine-tune the parameters of the machine learning model 550. The training process 500 can include feeding the training input 510 to the machine learning model 550. In some implementations, the training input 510 includes input from one or more users, such as new code. Upon receiving the training input 510, the machine learning model 550 can generate an output 515 (e.g., an autocomplete suggestion 125). The training input 510 can include some or all of the code library 114.

[0044] In some implementations, the output 515 is used by a loss function 530 to generate a loss 540. The loss function 530 compares the output 515 to the label 520 to generate the loss 540, where the loss 540 indicates the difference between the label 520 (i.e., the true value representation of the code) and the output 515 (i.e., the autocomplete suggestion). The loss function 530 can implement any suitable technique to determine the loss, such as regression loss, mean squared error, mean squared logarithmic error, mean absolute error, binary classification, binary cross entropy, hinge loss, multi-classification loss, etc. The loss 540 can then be fed directly to the machine learning model 550. Here, the machine learning model 550 processes the loss 540 and adjusts one or more parameters of the machine learning model 550 to explain the loss 540.

[0045] In some implementations, the training process 500 occurs in real time. In other words, a user may type input in a development environment that is received as training input 510. The machine learning model 550 may generate one or more auto-fill suggestions 125 (i.e., output 515) that are provided to the user 12 in response to the user input 120. The user 12 may then accept one of the auto-fill suggestions 125 or manually complete the user input 120. The resulting final code (i.e., output 515 or completion typed by the user 12) may be used to label 520 additional training input 510 for the machine learning model 550. The loss function 530 may then generate a loss 540 based on the label 520 and the output 515.

[0046] Figure 6 is a flow chart of an exemplary arrangement of operations of a method 600 for providing auto-fill suggestions for code completion in a development environment. The method 600 may be, for example, Figure 1 The method 600 may be executed by various elements of the system 100. For example, the method 600 may be executed on the data processing hardware 154 of the remote system 150, on the data processing hardware 112 of the user device 16, or some combination thereof. At operation 610, the method 600 includes obtaining, from the user interface 14 executed on the user device 16, a user input 120 representing source code generated within the development environment 112, the source code being created using a specific programming language and a programming code library 114. At operation 620, the method 600 includes determining, based on the user input 120, using the machine learning model 550, auto-fill suggestions 125. The auto-fill suggestions 125 continue the source code represented by the user input 120. At operation 630, the method 600 includes determining, based on the development environment 112 and the programming code library 114, whether the auto-fill suggestions 125 are semantically correct. At operation 640 , the method 600 includes transmitting the autofill suggestion 125 for display on the user interface 14 of the user device 16 when the autofill suggestion 125 is semantically correct.

[0047] Figure 7 700 that can be used to implement the systems and methods described in this document. Computing device 700 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The components shown here, their connections and relationships, and their functions are intended to be exemplary only and are not intended to limit implementations of the inventions described and / or claimed in this document.

[0048] The computing device 700 includes a processor 710, a memory 720, a storage device 730, a high-speed interface / controller 740 connected to the memory 720 and a high-speed expansion port 750, and a low-speed interface / controller 760 connected to a low-speed bus 770 and the storage device 730. Each of the components 710, 720, 730, 740, 750, and 760 is interconnected using various buses and can be installed on a common motherboard or installed in other ways as appropriate. The processor 710 can process instructions for execution within the computing device 700, including instructions stored in the memory 720 or on the storage device 730, to display graphical information of a graphical user interface (GUI) on an external input / output device (such as a display 780 coupled to the high-speed interface 740). In other implementations, multiple processors and / or multiple buses and multiple memories and multiple types of memories can be used as appropriate. In addition, multiple computing devices 700 can be connected (for example, as a server bank, a blade server group, or a multi-processor system), where each device provides a portion of the necessary operations.

[0049] The memory 720 stores information non-temporarily within the computing device 700. The memory 720 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-temporary memory 720 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by the computing device 700. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0050] The storage device 730 is capable of providing mass storage for the computing device 700. In some implementations, the storage device 730 is a computer-readable medium. In various implementations, the storage device 730 may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid-state memory device, or an array of devices (including devices in a storage area network or other configuration). In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable medium or a machine-readable medium, such as the memory 720, the storage device 730, or a memory on the processor 710.

[0051] The high-speed controller 740 manages bandwidth-intensive operations of the computing device 700, while the low-speed controller 760 manages less bandwidth-intensive operations. This division of responsibilities is exemplary only. In some implementations, the high-speed controller 740 is coupled to the memory 720, the display 780 (e.g., through a graphics processor or accelerator), and the high-speed expansion port 750 that can accept various expansion cards (not shown). In some implementations, the low-speed controller 760 is coupled to the storage device 730 and the low-speed expansion port 790. The low-speed expansion port 790, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner; or to a networking device, such as a switch or a router, for example, through a network adapter.

[0052] As shown, computing device 700 can be implemented in a variety of different forms. For example, it can be implemented as a standard server 700a or multiple implementations in a group of such servers 700a, implemented as a laptop computer 700b, or implemented as part of a rack server system 700c.

[0053] Various implementations of the systems and techniques described herein may be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpreted on a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device.

[0054] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0055] The processes and logic flows described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The processes and logic flows can also be performed by a dedicated logic circuit system, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). For example, processors suitable for executing computer programs include both general-purpose microprocessors and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as a magnetic disk, a magneto-optical disk, or an optical disk, or be operably coupled to receive data from one or more mass storage devices or transfer data to one or more mass storage devices, or both. However, a computer does not have to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or incorporated in special purpose logic circuitry.

[0056] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user and optionally a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0057] A software application (i.e., software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0058] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., instruction sequences) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0059] A variety of implementations have been described. However, it should be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Therefore, other implementations are within the scope of the following claims.

Claims

1. A computer-implemented method (600) which, when executed by data processing hardware (154), causes the data processing hardware (154) to perform operations comprising: obtaining user input (120) from a user interface (14) executing on a user device (16) representing source code generated within a development environment (112), the source code being created using a particular programming language and a programming code library (114); determining, using a machine learning model (550), autocomplete suggestions (125) based on the user input (120), the autocomplete suggestions (125) continuing the source code represented by the user input (120); Determining whether the autocomplete suggestion (125) is semantically correct based on the development environment (112) and the programming code base (114) using a rule-based semantic checker (250) configured for the specific programming language; as well as When the autofill suggestion (125) is semantically correct, the autofill suggestion (125) is transmitted for display on the user interface (14) of the user device (16).

2. The computer-implemented method (600) of claim 1, wherein determining whether the autocomplete suggestion (125) is semantically correct comprises comparing the autocomplete suggestion (125) to a structural representation of the programming code base (114).

3. The computer-implemented method (600) of claim 2, wherein the structural representation of the programming code library (114) is an abstract syntax tree.

4. The computer implemented method (600) of claim 2 or claim 3, wherein the structural representation of the programming code library (114) is stored in a memory cache (215).

5. The computer-implemented method (600) of any one of claims 1 to 4, wherein the machine learning model (550) and the rule-based semantic checker (250) are co-located within a cloud computing environment (150).

6. The computer-implemented method (600) of any one of claims 1 to 5, wherein the operations further comprise: determining a plurality of autofill suggestions (125) based on the user input (120) using the machine learning model (550); as well as For each respective autofill suggestion (125) in the plurality of autofill suggestions (125): using the rule-based semantic checker (250) to determine whether the corresponding autocomplete suggestion (125) is semantically correct based on the development environment (112) and the programming code base (114); and When the corresponding autofill suggestion (125) is semantically correct, the corresponding autofill suggestion (125) is transmitted for display on the user interface (14) of the user device (16).

7. The computer-implemented method (600) of claim 6, wherein the rule-based semantic checker (250) determines whether each respective one of the plurality of auto-fill suggestions (125) is semantically correct within an allocated time budget.

8. The computer-implemented method (600) of claim 7, wherein: When the rule-based semantic checker (250) is unable to determine within the allocated time budget whether each respective one of the plurality of auto-fill suggestions (125) is semantically correct, the operations further include: selecting a portion of the plurality of autofill suggestions (125) to maximize coverage within an allotted time budget; and For each respective autofill suggestion (125) of the portion of the plurality of autofill suggestions (125): using the rule-based semantic checker (250) to determine whether the corresponding autocomplete suggestion (125) is semantically correct based on the development environment (112) and the programming code base (114); and When the corresponding autofill suggestion (125) is semantically correct, the corresponding autofill suggestion (125) is transmitted for display on the user interface (14) of the user device (16).

9. The computer-implemented method (600) of any one of claims 6 to 8, wherein determining whether each respective one of the plurality of auto-fill suggestions (125) is semantically correct comprises: grouping the respective autofill suggestions (125) with similar autofill suggestions (125) from the plurality of autofill suggestions (125) into one or more groups of autofill suggestions (125); as well as An example autofill suggestion (125) from each of the one or more sets of autofill suggestions (125) is compared to a structural representation (300) of the programming code library (114).

10. The computer-implemented method (600) of any one of claims 1 to 9, wherein: When the autocomplete suggestion (125) is semantically incorrect, the operations further include: generating one or more constraints (225) using the rule-based semantic checker (250); and A new auto-fill suggestion (125) is determined based on the user input (120) and the one or more constraints (225) using the machine learning model (550).

11. The computer-implemented method (600) of any one of claims 1 to 10, wherein an integration model (260) facilitates communication between the machine learning model (550) and the rule-based semantic checker (250).

12. A system (100), comprising: Data processing hardware (154); as well as Memory hardware (156) in communication with the data processing hardware (154), the memory hardware (156) storing instructions that, when executed on the data processing hardware (154), cause the data processing hardware (154) to perform operations, the operations comprising: obtaining user input (120) from a user interface (14) executing on a user device (16) representing source code generated within a development environment (112), the source code being created using a particular programming language and a programming code library (114); determining, using a machine learning model (550), autocomplete suggestions (125) based on the user input (120), the autocomplete suggestions (125) continuing the source code represented by the user input (120); Determining whether the autocomplete suggestion (125) is semantically correct based on the development environment (112) and the programming code base (114) using a rule-based semantic checker (250) configured for the specific programming language; and When the autofill suggestion (125) is semantically correct, the autofill suggestion (125) is transmitted for display on the user interface (14) of the user device (16).

13. The system (100) of claim 12, wherein determining whether the autocomplete suggestion (125) is semantically correct comprises comparing the autocomplete suggestion (125) to a structural representation of the programming code base (114).

14. The system (100) of claim 13, wherein the structural representation of the programming code library (114) is an abstract syntax tree.

15. The system (100) of claim 13 or claim 14, wherein the structural representation of the programming code library (114) is stored in a memory cache (215).

16. The system (100) of any one of claims 12 to 15, wherein the machine learning model (550) and the rule-based semantic checker (250) are co-located in a cloud computing environment (150).

17. The system (100) of any one of claims 12 to 16, wherein the operations further comprise: determining a plurality of autofill suggestions (125) based on the user input (120) using the machine learning model (550); as well as For each respective autofill suggestion (125) in the plurality of autofill suggestions (125): using the rule-based semantic checker (250) to determine whether the corresponding autocomplete suggestion (125) is semantically correct based on the development environment (112) and the programming code base (114); and When the corresponding autofill suggestion (125) is semantically correct, the corresponding autofill suggestion (125) is transmitted for display on the user interface (14) of the user device (16).

18. The system (100) of claim 17, wherein the rule-based semantic checker (250) determines whether each respective one of the plurality of autofill suggestions (125) is semantically correct within an allotted time budget.

19. The system (100) of claim 18, wherein: When the rule-based semantic checker (250) is unable to determine within the allocated time budget whether each respective one of the plurality of auto-fill suggestions (125) is semantically correct, the operations further include: selecting a portion of the plurality of autofill suggestions (125) to maximize coverage within an allotted time budget; and For each respective autofill suggestion (125) of the portion of the plurality of autofill suggestions (125): using the rule-based semantic checker (250) to determine whether the corresponding autocomplete suggestion (125) is semantically correct based on the development environment (112) and the programming code base (114); and When the corresponding autofill suggestion (125) is semantically correct, the corresponding autofill suggestion (125) is transmitted for display on the user interface (14) of the user device (16).

20. The system (100) of any one of claims 17 to 19, wherein determining whether each respective one of the plurality of auto-fill suggestions (125) is semantically correct comprises: grouping the respective autofill suggestions (125) with similar autofill suggestions (125) from the plurality of autofill suggestions (125) into one or more groups of autofill suggestions (125); as well as An example autofill suggestion (125) from each of the one or more sets of autofill suggestions (125) is compared to a structural representation (300) of the programming code library (114).

21. The system (100) of any one of claims 12 to 20, wherein: When the autocomplete suggestion (125) is semantically incorrect, the operations further include: generating one or more constraints (225) using the rule-based semantic checker (250); and A new auto-fill suggestion (125) is determined based on the user input (120) and the one or more constraints (225) using the machine learning model (550).

22. The system (100) of any one of claims 12 to 21, wherein an integration model (260) facilitates communication between the machine learning model (550) and the rule-based semantic checker (250).

Images