Method for improving the technical process of computer programming using dynamic programming languages, computer program, computer (type inference in dynamic languages)

Abstract

To improve a technological process of programming a computer using a dynamic programming language.SOLUTION: The process comprises: a step of generating a first portion of training data which maps types in the dynamic programming language to corresponding functions and methods by performing information retrieval on documentation libraries in the dynamic programming language; or a step of generating a second portion of training data which maps program variables to the corresponding functions and methods by performing data flow analysis on a plurality of pre-existing programs written in the dynamic programming language; or combination of these. The process trains a neural network on the first portion or second portion or a combination thereof of training data to infer unknown types in the dynamic programming language. The process carries out inference with the trained neural network to infer the unknown types. The process facilitates programming in the dynamic programming language based on the inferred unknown types.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to electrical, electronic, and computer technologies, and more particularly to dynamic programming languages, machine learning, and the like. [Background technology]

[0002] Dynamic programming languages are a class of high-level programming languages that perform many common programming operations at run time that static programming languages perform at compile time, including, for example, extending a program by adding new code, extending objects and definitions, modifying the type system, etc. Summary of the Invention [Problem to be solved by the invention]

[0003] Many dynamic programming languages do not enforce types of any kind, but type inference is important for preventing bugs, improving program comprehension, and improving automation through code generation. [Means for solving the problem]

[0004] The present principles provide techniques for type inference in dynamic languages. In one aspect, an exemplary method for improving the technical process of computer programming using a dynamic programming language comprises generating a first portion of training data mapping types in the dynamic programming language to corresponding functions and methods in the dynamic programming language by performing an information lookup on a documentation library in the dynamic programming language, training a neural network on the first portion of training data to infer unknown types in the dynamic programming language, performing inference using the trained neural network to infer the unknown types of program variables in samples of code written in the dynamic programming language by inferring the unknown types based on the existence of the corresponding functions and methods, and facilitating programming in the dynamic programming language based on the inferred unknown types.

[0005] In another aspect, another exemplary method for improving a technical process for computer programming with a dynamic programming language comprises generating training data mapping types in the dynamic programming language to corresponding functions and methods in the dynamic programming language, training a neural network with at least the training data to infer unknown types in the dynamic programming language, reducing a number of types to search during inference based on the mapping of program variables, performing the inference using the trained neural network to infer the unknown types, and facilitating programming in the dynamic programming language based on the inferred unknown types.

[0006] In yet another aspect, an exemplary computer comprises a memory embodying computer-executable instructions and at least one processor coupled to the memory and operated by the computer-executable instructions to perform a method for improving a technological process of programming the computer with a dynamic programming language, the method comprising: generating a first portion of training data mapping types in the dynamic programming language to corresponding functions and methods in the dynamic programming language by instantiating a neural network and a search engine and using at least the search engine to perform an information search on a documentation library in the dynamic programming language, training a neural network on the first portion of training data to infer unknown types in the dynamic programming language, performing inference using the trained neural network to infer the unknown types of program variables in samples of code written in the dynamic programming language by inferring the unknown types based on the existence of the corresponding functions and methods, and facilitating programming in the dynamic programming language based on the inferred unknown types.

[0007] In a further aspect, another exemplary computer comprises a memory embodying computer-executable instructions and at least one processor, coupled to the memory and operated by the computer-executable instructions, for performing a method for improving a technological process of programming the computer with a dynamic programming language, the method comprising: instantiating a neural network and a static software analysis tool; and performing the data flow analysis on the plurality of existing programs written in the dynamic programming language to generate training data mapping program variables in the dynamic programming language to functions and methods of the dynamic programming language; training a neural network with at least the training data to infer unknown types in the dynamic programming language; reducing a number of types to search during inference based on the mapping of program variables; performing the inference using the trained neural network to infer the unknown types; and facilitating programming in the dynamic programming language based on the inferred unknown types.

[0008] As used herein, "facilitating" an action includes performing the action, making the action easier, assisting in the performance of an action, or having an action performed. Thus, by way of example and not limitation, instructions executing on one processor may facilitate an action performed by instructions executing on a remote processor by sending appropriate data or commands to cause or assist in the performance of the action. For the avoidance of doubt, even if an actor facilitates an action by other than performing the action, the action is nevertheless performed by some entity or combination of entities.

[0009] One or more embodiments of the present invention, or elements thereof, can be implemented in the form of a computer program product including a computer-readable storage medium with computer-usable program code for performing the illustrated method steps. Furthermore, one or more embodiments of the present invention, or elements thereof, can be implemented in the form of a system (or apparatus) including a memory and at least one processor coupled to the memory and operative to perform the illustrated method steps. Still further, in another aspect, one or more embodiments of the present invention, or elements thereof, can be implemented in the form of a means for performing one or more of the method steps described herein, which may include (i) a hardware module, (ii) a software module stored on a computer-readable storage medium (or multiple such media) and implemented on a hardware processor, or (iii) a combination of (i) and (ii), any of which implements specific techniques described herein.

[0010] The techniques of the present invention can provide significant beneficial technical advantages. Some embodiments may not have these potential advantages, and these potential advantages are not necessarily required for all embodiments. For example, one or more embodiments may provide one or more of the following:

[0011] To improve the technical process of programming in dynamic programming languages by providing reasonably high quality labeled data at scale for dynamic programming languages that outperforms static type checkers.

[0012] High-quality labeled data enables better probabilistic type inference systems. These and other features and advantages of the present invention will become apparent from the following detailed description of illustrative embodiments, which is to be read in connection with the accompanying drawings. [Brief explanation of the drawings]

[0013] [Figure 1]FIG. 1 illustrates type prediction according to the prior art.

[0014] [Figure 2] FIG. 1 illustrates exemplary code and documentation analyzed using an aspect of the present invention.

[0015] [Figure 3] FIG. 1 illustrates an example of a class with aliases to different classes that can be successfully analyzed using an aspect of the present invention.

[0016] [Figure 4] FIG. 1 illustrates the distribution of classes in the top 25 modules of the dataset used to test embodiments of the present invention.

[0017] [Figure 5] FIG. 1 illustrates a document string extraction algorithm according to an aspect of the present invention.

[0018] [Figure 6] FIG. 3 illustrates a data flow diagram for the first script in FIG. 2 according to one embodiment of the present invention.

[0019] [Figure 7] FIG. 3 illustrates an internal representation for the first script in FIG. 2 according to one embodiment of the present invention.

[0020] [Figure 8] FIG. 10 illustrates an example of sharing a few methods related to the duck typing aspect of the present invention.

[0021] [Figure 9] FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention. [Figure 10] FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention. [Figure 11]FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention. [Figure 12] FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention. [Figure 13] FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention. [Figure 14] FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention. [Figure 15] FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention. [Figure 16] FIG. 10 illustrates exemplary test results achieved using an exemplary embodiment of the present invention.

[0022] [Figure 17] FIG. 1 shows a bar graph comparing the percentage of predicted types for two prior art techniques and an exemplary embodiment of the present invention.

[0023] [Figure 18] FIG. 2 is a diagram similar to FIG. 1, but showing type predictions obtained using an exemplary embodiment of the present invention.

[0024] [Figure 19] FIG. 1 illustrates an overview of types inferred by document strings and static analysis techniques used in aspects of the present invention.

[0025] [Figure 20] FIG. 20 illustrates the distribution of top modules whose types were inferred by the two methods of FIG. 19, according to one embodiment of the present invention.

[0026] [Figure 21] FIG. 2 illustrates an exemplary type inference stage, according to an aspect of the present invention. [Figure 22] FIG. 2 illustrates an exemplary type inference stage, according to an aspect of the present invention.

[0027] [Figure 23] FIG. 1 illustrates a cloud computing environment according to one embodiment of the present invention.

[0028] [Figure 24] FIG. 2 illustrates abstraction model layers according to one embodiment of the present invention.

[0029] [Figure 25] 1 illustrates a computer system useful for implementing one or more aspects or elements, or combinations thereof, of the present invention, and is representative of a cloud computing node according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0030] In recent years, dynamically typed languages such as the PYTHON® language (a registered trademark of the PYTHON SOFTWARE FOUNDATION, Beaverton, Oregon, USA) have become ever more popular. The PYTHON language is well known to those skilled in the art. While these languages reduce the need for mandatory type annotations, types still play a relevant role in program understanding and preventing runtime errors. An attractive option is to automatically infer types to obtain static guarantees without writing types. Existing inference techniques mostly rely on static typing tools such as PyType to perform direct type inference. Recently, neural type inference has been proposed. However, neural type inference requires data and therefore relies on the collection of labeled data based on static typing tools. However, such tools (i.e., static typing tools such as PyType) are poor at inferring user-defined types. Furthermore, these languages have very little developer-initiated type annotation. One or more embodiments provide novel techniques for generating high-quality types that advantageously utilize 1) information retrieval techniques operating on well-documented libraries to extract types, or 2) usage patterns by analyzing large repositories of programs, or a combination thereof. Our experimental results demonstrate that these techniques are more accurate than prior art, address weaknesses of static tools, and / or may be useful for generating large-scale labeled datasets for type inference using machine learning methods. F1 scores (harmonic mean of precision and recall) range from 0.52 to 0.58 for exemplary embodiments, compared to 0.06 for static typing tools. Our experiments used them to generate over 37,000 types for over 700 modules. Note that a "well-documented" library, as used herein, is one in which the usage of its various methods and classes is described in natural language. This documentation typically appears within the code itself, defining the purpose of functions / classes and their corresponding input / output formats.

[0031] One or more embodiments may find practical applications in, for example, dynamically typed languages, type inference, static analysis, PYTHON, big code, software repository mining, and the like.

[0032] As mentioned above, dynamically typed languages such as PYTHON are gaining popularity. One reason for this is the unprecedented growth of artificial intelligence (AI) and the widespread adoption of PYTHON in AI frameworks. Like many dynamic programming languages, PYTHON does not statically enforce types and only detects errors at runtime. This allows programmers to quickly build prototypes, making it popular. However, types can help understand programs, detect errors early, and improve program correctness. PYTHON introduced optional type declarations, but their adoption has so far been low. Furthermore, traditional type inference has proven largely ineffective. Figure 1 shows the prediction results of PyType on a dataset of 408 repositories. Current techniques frequently generate "Any" as a type, which is equivalent to no information. Furthermore, only 14% of generated types are user-defined or library types, which tend to be more prevalent in user code.

[0033] In this context, machine learning is a promising approach, and recent systems using neural networks to perform type inference have emerged. However, learning approaches require large amounts of type-annotated code for training, which currently does not exist. In fact, neural systems currently rely on tools such as PyType and Mypy software (available from The Mypy Project), or user-specified annotations for a gold standard. This labeled data may be distorted in ways that affect the quality of the models built and, when used as a gold standard, may provide misleading accuracy estimates.

[0034] One or more embodiments provide techniques for generating high-quality types for methods that can serve as labeled data for data-hungry machine learning approaches. While machine learning techniques show promise for the general problem of type inference, the current state of the art does not provide adequate mechanisms for addressing the problem of obtaining good-quality labeled data. Advantageously, one or more embodiments (a) extract high-quality types from well-documented framework code for inferring types, or (b) extract high-quality types from their use in code available for analysis, or a combination thereof. Advantageously, one or more embodiments generate better datasets for use in building improved probabilistic type inference systems.

[0035] One or more embodiments start with framework data because it is commonly used and well-documented. To infer types from documentation, one or more embodiments use information retrieval techniques to collect possible types specified in the documentation and map them to a set of classes that are indexed using aspects of the present invention. To infer types from usage, one or more embodiments mine usage from a large number (e.g., millions) of programs and explore duck typing based on program analysis. Duck typing is the idea that if a class contains all the methods called on a given object, it is a likely candidate for that object's type. While the idea of duck typing itself is not new, one or more embodiments apply it in a novel way. Specifically, one or more embodiments analyze a large (e.g., millions) repository of code files and combine the use of the same libraries across these files. While one or more embodiments analyze individual programs, one or more embodiments observe how data flows from common API (application programming interface) calls across all programs to the objects returned by the calls. The duck typing method advantageously enables the inference of user-defined types. Prior art duck typing techniques cannot generally infer primitive or built-in types because they do not correspond to known classes. In one or more embodiments, type inference from documentation advantageously allows for better performance for built-ins and primitives. We have discovered that combining these techniques overcomes the weaknesses of each technique and allows for a wider variety of types to be inferred.

[0036] One or more embodiments advantageously address one or more of the following:

[0037] Using these two techniques to obtain types that are accurate enough to provide high-quality labeled data.

[0038] Using these two techniques to obtain labeled type inference data in PYTHON addresses some of the weaknesses of current state-of-the-art tools such as PyType.

[0039] Using these two techniques to obtain types provides a set of distinct types, and the union of the two approaches increases the size of the labeled set.

[0040] Our experiments show that in one or more exemplary embodiments, the two techniques were able to generate types for over 37,000 methods and functions in 756 Python modules. Our experiments compared the precision and recall of type inference techniques according to aspects of the present invention with a set of types inferred from dynamic techniques and manual annotations for a sample size of over 200 functions. Our F1 scores for static analysis and documentation inference were 0.52 and 0.58, respectively, compared to 0.06 for PyType. Furthermore, state-of-the-art neural prediction systems, such as TypeWriter, achieve 0.64 with trained models based on type annotations. One or more embodiments are fully unsupervised and advantageously generate labeled data useful for building better neural models for type inference.

[0041] [example]

[0042] Figure 2 illustrates the ideas behind the large-scale generation of labeled types for API calls, for example, in the form of code and documentation for read_csv. Example Script 1, labeled 101, calls a function that reads a pandas.Dataframe object from the pandas library, passes its return value to the function, and uses that object as a receiver for the drop and drop_na calls. Example Script 2, labeled 103, has a more direct relationship between the read_csv and to_csv calls on the returned object. From a duck typing perspective across multiple scripts, we can see that the type of the object returned from the read_csv call should support drop, drop_na, head, and to_csv. From the type definitions of the API's classes, pandas.Dataframe is identified as a candidate class. Figure 2 also shows the documentation associated with the read_csv function in 105. We can see that the documentation is not well-formed enough to clearly indicate the referenced class, so additional processing is appropriate to infer possible types. The classes are mentioned informally, with no reference to their fully qualified names. Because natural language references use phrases such as DataFrame or Text Parser, some mechanism must be provided to resolve the two classes mentioned here to their fully qualified names (e.g., pandas.core.frame.DataFrame). In one or more embodiments, simple information retrieval techniques can be used to find potential types from the documentation.

[0043] [Dataset]

[0044] Our experiments used a publicly available dataset containing over one million PYTHON programs. To collect related classes and methods, we identified the top 500 imported modules in these PYTHON programs. For each of these modules, we programmatically created a virtual environment, installed the module using pip, and then used the PYTHON inspect API to collect all classes, their methods, and associated documentation strings within the loaded module. The PYTHON introspect API not only provides classes from loaded modules, but can also collect classes from modules in the dependency tree of the loaded module. Furthermore, a quirk of the PYTHON inspect API is that it may specify many classes with aliases to the same class based on module dependencies. The table in Figure 3 (example of classes with aliases to different classes) shows such an example. The first two DataFrame classes in statsmodels actually map to classes in a completely different module, pandas. Furthermore, due to PYTHON packaging, multiple PYTHON classes within a module appear with different qualified names (as the bokeh classes indicate).

[0045] Starting with a seed set of 500 modules, we obtained a result set of 1,017 modules, 167,872 classes, and 164,134 functions. To cleanse the dataset, we loaded each of the 167,872 classes returned by the inspect API into a virtual environment, loaded the class using the name returned by the API, and noted the actual name when printing the string representation of the class. We derived a mapping table between classes and their actual aliased classes, as shown in the table in Figure 3, resulting in 92,277 unique classes after aliasing. A similar approach was taken for function name aliases, resulting in 164,134 function names being loaded and aliased to 91,818 functions.

[0046] Figure 4 (Class Distribution of Top 25 Modules) illustrates the class distribution of the top 25 modules. As shown, modules with diverse functionality are covered, ranging from visualization libraries (e.g., plotly) to cloud management libraries (e.g., kubernetes) to data science libraries (e.g., sklearn and pandas). In total, we provided 26,800 class methods and 53,441 functions with documentation strings.

[0047] [Type inference from documentation strings]

[0048] Type Extraction: As shown in Figure 2, API library documentation is typically well-structured and tends to be written using rich structured text, allowing documentation generation from packages such as Sphinx. One relevant challenge is how to leverage information retrieval techniques to infer type information from such documentation. In our experiments, we focused on returns to illustrate an example method, as described in Algorithm 1 (docstring extraction algorithm) in Figure 5. Given a set l of modules, we collect all functions and methods declared in the module in l. For each f, we collect its class (if it is a method) in set C and obtain the corresponding docstring r. In our experiments, we use the sphinx library in PYTHON to parse the docstrings into reconstructed text. In our example, we remove the "return" portion from the entire docstring of the method, resulting in the text shown in Figure 2. This structured text r informally describes the return value of each class and function. For example, Figure 2 describes the return value as a DataFrame or TextParser. To infer qualified types, we create a "document" d by setting function and content fields for each function or method, and index d into the search and analytics engine's text index. At the end of all module inspections, the return type of each method is added to the index. We then loop through all classes in C and search the search and analytics engine's index to find all documents for d that mention this class in their return text. Each d contains an inferred type set, where the fully qualified class name c is appended to the function's return type. At the end of the extraction process, all d in the index are functions whose types have been inferred based on their documentation strings if d contains an inferred type field set.

[0049] Cleansing: Because the type inferences from this mechanism are very noisy, our experiments employ a post-processing stage to filter out erroneous annotations. Specifically, on the list of methods and their inferred types returned by the above stage, we perform the following:

[0050] Using the mapping between classes and their actual alias classes (see the table in Figure 3), each returned user-defined type was mapped to the correct alias. For example, the class pandas.DataFrame is mapped to pandas.core.frame.DataFrame. Note that both forms are valid in Python, and in fact user code frequently contains imports of pandas.DataFrame, but at runtime the interpreter returns pandas.core.frame.DataFrame.

[0051] Removes any types that cannot be resolved to any valid type based on the classes provided by the inspect API, but which would fail to load at runtime because they do not exist.

[0052] User-defined types from different libraries are eliminated if there is a class in the same library that is a candidate return type for that type. This last approach is based on the heuristic that if a class with the same name exists in the same library, it is more likely to be a candidate return type than a class with the same name in another library. Note that existing type inference systems, such as TypeWriter, ignore the fully qualified name of a class, which is difficult to solve and was a problem in our experiments.

[0053] If a built-in or primitive is matched, remove all other classes. This step is appropriate to avoid matching classes that have the same name as a built-in or primitive (e.g., Dict) but are unlikely to be a clear match.

[0054] [Type inference by analysis]

[0055] One way to infer types is to run data flows over millions of scripts and observe which methods are invoked on the objects returned by particular method calls. One or more embodiments provide a set of new changes introduced to the static analysis infrastructure to support this type analysis, or techniques for performing duck typing in an improved manner, or a combination thereof.

[0056] Extended Analysis Approach: To perform this data flow, our experiments limited the scope of analysis to the level of each Python file in our sample set. We used each method in the script as a starting point, ensuring maximum coverage of the code within the script, including the script itself. Our exemplary analysis was inter-procedural, so as shown in Figure 2, 101, the data flow follows the procedure massage_data, finding that the return value of pandas.read_csv has both dropna and drop called on it, followed by a call to head guarded by a condition.

[0057] Those skilled in the art will appreciate that PYTHON scripts are generally not self-contained but contain library and API call imports or user modules with code contained in other files. To analyze a large number of files in such a situation, we have found it important not to assume that stubs for such calls can be created en masse or that the library code is analyzable. One or more embodiments provide a mechanism called a "turtle" to handle such imports or calls to functions that were not part of the script. In this exemplary embodiment, all returns from API calls are represented as instances of a single "turtle" type (a minimal model of the objects returned by the library), and all calls to such objects return new instances of that type. Similarly, accessing properties of those objects returns the object itself. This can be expressed in a general analysis framework and formalism, allowing for customization in three aspects of the analysis. Consider these three aspects in terms of analysis abstractions that should be customized for any analysis framework.

[0058] Overall, there are three important changes that make the analysis framework suitable for enabling the analysis of turtle-based programs:

[0059] Importing required APIs should be replaced by turtle creation. How import calls are represented varies depending on the analysis framework. In the example implementation, import calls are modeled as calls to a composed function that returns a newly allocated object of type "turtle". This function is analyzed using call site sensitivity, i.e., a separate analysis for each call, so that a different turtle object is created for each API import. In Figure 2, read_csv is imported, so the return of that call is represented by a turtle.

[0060] The semantics of property reads should be changed so that any property read on a turtle returns the container object itself. This can be modeled by doing a field-insensitive analysis on objects of type turtle, i.e., modeling all of their properties as a single property. When a turtle object is created, it is assigned to that single property.

[0061] The semantics of function calls should be enforced so that every call on an object of type turtle is a call to a composed function that returns a new turtle object. For function calls, all functions can be modeled with the same composed function that simply returns a new turtle. In Python, a call like pd.read_csv would be like first doing a property read and then a call. Since a property read on a turtle already returns the same object, the function call composition model is sufficient for method calls as well.

[0062] The extended analysis framework combines standard call graph construction with pointer analysis to calculate, for each call site, which functions may be called and, for each value, which object it may hold. The analysis starts from the root function, analyzes each instruction, and adds functions found at the call site to a work queue. To make the analysis more concrete, we use the internal representation (IR) of script 101 in Figure 2. The code is structured as a sequence of basic blocks (denoted as BB0, BB1, etc. in Figure 7) of operations such as property reads, and all values are static single assignments. We will explain step-by-step how the analysis of turtles is performed and what analysis is provided when analyzing a script. Figure 6 shows the data flow diagram for script 1 101 in Figure 2. In Figure 6, circles 1, 2, 3, 4, and 5 represent turtles, and arrows represent data flow between turtles. In circle 1, "pandas" is unknown, and logic flows to the next circle. The results of pandas.read_csv are shown for dropna, drop, and head. Receiver 0 means that each circle can have multiple arguments, like a function call. Dropna is called with 2 (pandas.read_csv), so that is the receiver. Figure 7 shows the internal representation for Script 1 101 in Figure 2, on which the analysis is computed. "invokeFunction" represents a turtle call, "getfield" represents a read, "putfield" represents a write, and "invokestatic" in the second line is a special function call used to represent an import. "BB" stands for basic block, and "gt" determines whether one value is greater than another. Given the teachings herein, one skilled in the art will understand the IR in Figure 7 based on their understanding of compilers.

[0063] Instruction 2 is the import corresponding to line 1 of location 101 in Figure 2. This instruction assigns the imported script to v40, represented by the turtle t1.

[0064] Instructions 3-5 create the inner function massage_data, which is shown in lines 3 through 8. Functions can be first-class, so they are represented as objects in the analysis.

[0065] In command 6, the property read_csv is read from v40, which holds the imported pandas script, and assigned to v47, which is also t1.

[0066] Instruction 7 calls v47 as a function. v47 holds t1, and since the semantics of a function call on a turtle is to create a new turtle, it assigns a new turtle, t2, to v46.

[0067] With the exception of one, the remaining instructions are mostly analog.

[0068] Instruction 9 calls v44, which is massage_data. Since this is not a turtle, the function's code is added to the analysis work queue. v46 is passed as an argument, which corresponds to passing the result of read_csv.

[0069] There is one aspect of the analysis that is not explained in this code snippet: at line 12 of location 101 in Figure 2, the built-in len call is passed the turtles returned by read_csv and ultimately massage_data. Because the analysis makes no assumptions about the meaning of turtles, calls to primitives are treated as simply returning any of the turtles used as arguments.

[0070] Duck Typing: As mentioned above, the analysis in one or more embodiments is neither sound nor complete. Traditional approaches to duck typing require that for every object O returned by a method call M, a set of method calls on O, called F, be observed. F must be defined for a given class C in order to infer M as its return type. Due to imprecision in the analysis, there may be erroneous methods in F. For example, in Figure 2, the call to head may not be called because it is under an if. This code works even if the small table returns a type that does not support head. To handle this situation, we approximate duck typing by instead calculating the size of F∪D, where D is the set of all methods defined for C. The likelihood of correct type inference is governed by two factors: (a) the size of F∪D, and (b) the number of possible type classes for a given method's return value. As (a) becomes larger, the type inference becomes more reliable. However, the combination of (b) a small number of classes and (a) a small number of shared methods can still sometimes imply valid inference. As an example of such a case, as shown in the table in Figure 8 (example of sharing a small number of methods), for example, pandas.array correctly returns pandas.core.arrays.base.ExtensionArray, and in fact, pandas.core.arrays.sparse.array.SparseArray turns out to be a subclass of pandas.core.arrays.base.ExtensionArray.

[0071] Analysis Cleansing: We find that initial duck typing of code often results in a large number of spurious types, and in one or more embodiments, we filter them in a series of stages:

[0072] Because duck typing in one or more embodiments is not completely accurate, the first step in one or more examples is to filter candidate types to those that match the maximum number of methods called in the code.

[0073] There are often many concrete types that share a common supertype within a set of types, in which case the subtypes are removed because they are covered by the supertype.

[0074] It may happen that most types in a set share a supertype S that is not itself in the set. In this case, types that are not subtypes of S are removed, as this is often due to imprecision in the analysis.

[0075] Use the list of functions and classes to remove items that are in fact modules but appear ambiguous due to the fact that the imports can be anything.

[0076] Eliminate classes and functions that are not valid as usual and use their aliases.

[0077] [Experimental evaluation]

[0078] Labeled Type Accuracy - Evaluation Against Dynamic Types: To develop a gold standard for evaluation, our experiments collected a set of types by observing runtime types. From a set of 408 repositories, we experimented with five repositories that (a) use pytest for unit testing and (b) were deemed relatively easy to set up without a set of additional dependencies on databases, servers, etc. For each function called by pytest in a test, a wrapper function was inserted that recorded its return type before returning. So-called "monkey patching" (i.e., locally extending or modifying supporting system software (affecting only the running instance of the program)) was leveraged by pytest and pytest fixtures to insert the wrappers. Those skilled in the art should be familiar with monkey patching, for example, from "Monkeypatching / mocking modules and environments," available at PyTest dot org (where "dot" replaces "." to avoid inclusion of code available in the browser), Holger Krekel and pytest-dev team, 2015-2020. The table in Figure 9 shows the number of tests passed or failed for each package (a summary of the number of passed and failed tests for each module and the number of methods inferred). The types collected by monkey patching are always sound, but not necessarily complete. We collected 2284 different methods corresponding to which types were available.

[0079] Each method was often annotated with multiple types. In some cases we manually inspected it and, where possible, augmented the set of dynamic types based on the documentation and ran the code. Proper care must be taken when instrumenting libraries.

[0080] The table in Figure 10 (statistics on discovered dynamic types) shows the total number of tests performed to obtain the 2,284 methods collected from dynamic typing. The number of matches for each type inference technique is very low, but our investigation shows that this is because the runtime often receives method names of functions stripped of their class names (e.g., obtained using dill). Of the three methods, extraction from docstrings was able to obtain types for most functions, followed by analysis and then PyType. Combining docstrings with analysis yielded return types for 203 methods, representing 9% of the methods for which dynamic information was available.

[0081] The table in Figure 11 (Precision for docstring and analysis-based type inference vs. PyType) shows the precision, recall for PyType, docstring-based type inference, and type inference based on static analysis and duck typing separately. While the F1 score for PyType is quite surprisingly low (.067), this result is consistent with the % precision reported in prior art when types are user-defined. Meanwhile, the F1 score for docstring-based type inference is 0.587, compared to 0.517 for static analysis, a significant improvement over PyType.

[0082] Accuracy of Labeled Types - Evaluating Class Constructors: Dynamic typing is one way to analyze the accuracy of type inference. One or more embodiments utilize a feature of the PYTHON language as a kind of sanity test for the accuracy of static analysis-based type inference. In PYTHON, as in many dynamic languages, constructors are simply another way. This fact was used to create a gold standard for how return types are known. A total of 92,277 classes were collected from inspect, and an exemplary embodiment of static analysis-based type inference was used to examine whether constructors were correctly inferred. Constructor recall was 0.0459, indicating that only a small fraction of the classes were actually used. Of these, static analysis-based duck typing generated correct types for 4,236 types and incorrect values for 130 types, resulting in an accuracy of 0.97. This error was due to an error in collecting class definitions. As an example, QtNetwork.QLocalSocket is a class whose usage is shown, and the code calls a method called waitForConnected. However, the inspect output does not find a method called waitForConnected, and therefore it is not associated with any class. Note that in general there are some imprecision in PYTHON's inspect API, which adds noise to the process. Nevertheless, tests with class constructors suggest that the analysis and duck typing approach is effective.

[0083] Labeled Type Accuracy - Manual Annotation: To further evaluate type inference for the two techniques, we selected a random sample of methods for each technique and attempted to manually evaluate whether the returned types were correct. Note that in this case we cannot actually evaluate recall or F1, but such qualitative evaluations are useful for understanding where each method's weaknesses lie. For analysis, our experiments sought to find as much information as possible from web documentation or inspection to determine whether the returned types were correct.

[0084] Static analysis sample: For 25 / 108 methods, we found insufficient documentation to correctly infer the return type. For the remaining methods, multiple types were often returned. Across all of these returned types, the example implementation was correct in 71 / 163 (43.56%) cases (each case reflecting a specific type inference), which is lower than what we observed with dynamic typing and may simply reflect sampling noise. One observation from this exercise is that we often find classes that are conceptually very similar but unrelated from a type perspective. As an example, we found scipy.spatial.kdtree.KDTree as the return type for sklearn.neighborhood.BallTree. While both are conceptually related and both derive from BinaryTree, one cannot naturally be substituted for the other. This is a weakness of duck typing approaches in general.

[0085] Docstring sample: We created another random sample of 200 methods from docstring type annotations. In our experiments, we were unable to manually check the return types of 67 methods, which were setter functions internal to libraries like plotly. Otherwise, our method correctly predicted the return type in 103 / 133 (77%) cases. One common issue with docstring-based types is their inaccuracy when documentation is insufficient or ambiguous. For example, with numpy, the documentation frequently states that the return value is an array, but what was returned was numpy.ndarray. In these cases, we can rely on usage patterns to infer a better type.

[0086] Weaknesses of Static Typing in PyType: The next task was to evaluate whether one or more exemplary embodiments for type inference address some of the weaknesses mentioned in the introduction with static typing tools like PyType. The comparison with PyType was chosen due to the observation that PyType performs slightly better than MYPY at type inference.

[0087] Figure 17 (Percentage of Predicted Types Using Dynamic Typing, PyType, and Exemplary Embodiments) illustrates the distribution of types under dynamic typing versus PyType and exemplary embodiments. Again, as discussed with respect to Figure 1, PyType tends to generate fewer user-defined types and generates a larger percentage of types labeled Any, which provides less accurate type information. One or more embodiments are biased toward void types, except when inferred from documentation. For purposes of harvesting high-quality labeled data, correctly modeling "void" is less important. For all other categories, there are exemplary embodiments that show dynamic typing inferring as many types as it generates.

[0088] To investigate the nature of each typing method and its errors for dynamic types, we computed a confusion matrix for each method. The table in Figure 12 (PyType confusion matrix for dynamic types) shows the same behavior observed for PyType as in Figure 1. Note that the system's tendency to answer "Any" applies to all types, but worsens for user-defined types, with no user-defined classes being inferred correctly. In fact, PyType frequently returned the name of the module (e.g., a user-defined class such as sympy.core.power.Pow is sympy). The inference techniques employed had the opposite bias. The table in Figure 13 (Confusion matrix for static analysis - the number of classes reflects the error (correct answer in brackets)) shows that the analysis tended to favor providing user-defined types. The built-in confusion reflects poor flow modeling - retrieving objects from tuples or lists and invoking methods on them incorrectly assumed a direct data flow. The table in Figure 14 (Confusion Matrix for Docstrings—Class Numbers Reflect Errors (Correct Answers in Parentheses)) shows the confusion matrix for docstrings, which also shows a similar error pattern to the analysis: primitives and built-ins are frequently confused with user-defined types. Most of these errors in the case of docstrings stem from the fact that the docstrings for numpy methods frequently return the user-defined class numpy.bool_ but state that they return a bool type. Similarly, for built-ins, they return a tuple, but when the documentation states the type returned by the tuple, in the example embodiment, the return type was incorrectly stated to be one of the mentioned types.

[0089] In our experiments, we also investigated the degree of agreement between the example implementation and PyType for the types returned by static analysis, as shown in the table in Figure 15. As shown in the table in Figure 15 (Confusion Matrix for PyType vs. Static Analysis—Number of Classes Reflects Differences (Agreement in Parentheses)), even when PyType returns a type that appears to be a class, the agreement is small at 22%. This agreement is far worse than the agreement between type inference from documentation strings and type inference from analysis (61%). Because PyType often does not return fully qualified class names, our experiments adjusted to measure overlap across 47 cases to account for cases where class names matched. In 36 cases, PyType returned a module as the returned type. This means that in 36 / 209 cases (17%), PyType returned inaccurate information about types when inferring classes.

[0090] A similar comparison with docstring-based type inference is shown in the table in Figure 16 (PyType vs. docstring confusion matrix - number of classes reflects differences (match in parentheses)). When PyType generated a class, it matched the docstring-based class in 47 cases, for a match of about 38%. Docstring-based inference often disagrees with PyType, especially for built-ins, likely because the documentation often refers to both the data structure and the type held within it (e.g., list of int). Again, PyType returned a module rather than a class 34 times, for a score of 34 / 124 (27%).

[0091] Properties of the Inferred Types Dataset: The table in Figure 19 (Summary of Types Inferred by the Two Methods) provides some summary statistics for the two methods of type inference. As the table shows, the two techniques together yield over 37,000 labeled types. Because their respective focuses are quite different, the degree of intersection between the two is small (410). Furthermore, when types were generated using the same method, agreement was achieved in 249 / 410 cases (61%). Figure 18 (Static Analysis and Docstrings-Based Predictions by Type Category) shows the distribution of predictions by type category. Comparing with PyType (see Figure 1), we can see that one or more embodiments complement PyType by generating more user-defined types instead of the None and Any types found in PyType. This, along with the accuracy results shown above, demonstrates that one or more embodiments can indeed generate higher-quality type annotations (turning indeterministic types such as Any and None into actual types), which can further improve existing type inference techniques.

[0092] Figure 20 shows the distribution of inferences for the top 25 modules for which types were inferred by the two methods. Some of these modules had the most classes, as shown in Figure 5, but not all of them. This is consistent with the fact that type inference by documentation strings and type inference by analysis have different strengths, and therefore modules with a large number of classes do not completely dominate the effect of type inference.

[0093] It will thus be appreciated that for PYTHON and similar dynamic languages, by leveraging documentation as well as usage information, reasonably high-quality labeled data can be generated on a large scale. One or more exemplary embodiments achieve significantly better performance than static type checkers and also generate high-quality labeled data, enabling better probabilistic type inference systems.

[0094] Many dynamic programming languages, such as Python, do not enforce any kind of type; that is, types are not mandated in these languages. Type inference is important for preventing bugs, improving program understanding, automating code generation, and determining where errors occur. Each program has an embedded context that helps type the returned object. This is known as "duck typing." The idea is that if a class contains all the methods that are called on a given object, it is a candidate type. This works well when a given program calls many methods on objects, but not so well otherwise. One or more embodiments employ duck typing across large code repositories to (a) infer types on a given program with a significant number of method calls and extrapolate to others, or (b) infer types using multiple method calls across programs and infer types across a collection of programs with the same API calls, or a combination thereof. In one or more embodiments, all methods in a class are collected and the return types of the functions are observed. Referring again to the discussion above regarding type extraction, As a non-limiting example, consider some random Python code and object, and suppose the lat and long methods are called on the object. Determine the types that have the lat and long methods as part of their location. You can assume that unknown items must be of type lat or long. If there are six methods that can be called on the object, then the object must support those six methods. That information can be used to filter what objects exist.

[0095] 21 and 22, one or more embodiments provide a technique for inferring types in dynamic programming languages. For example, given a set of programs S and a database D of all classes and methods for each class, as seen in input 2201, at 2202, for each program P in the set of programs S, a static analysis is applied that assumes that a call (I) to any imported library L in the given program P (here, a pd.read_csv object) returns a new object A of unknown type. In this example, head and dropna are called on the new object A of unknown type. At 2203, a dataflow analysis of P is used to identify all subsequent method calls on the new object A and determine the union of all methods M called on object A. These should be methods that any class conforming to that object must have. At 2204, for each call I in the imported library L, let F be a function L1 based on the analysis. At 2205, for the exact same function F across the entire set of programs S, a union of methods M' is created that combines all Ms across all Ps. In 2206, a database of classes and methods D is consulted to find a class C that contains all methods of M' for a given function F. In 2207, a mapping of F's function signature (function return type) to C is added. In 2208, the F → C mapping is applied to a new program containing a call I with a corresponding function F to infer the type I returns. Thus, since a machine learning process has been performed, it can be used in inference for new programs in step 2208. In some embodiments, type inference based on static analysis is implemented. In some cases, type inference can also be performed based on communication strings. For example, one can take the documentation associated with a particular API and infer the return type and parameter types based on the classes mentioned. There is a database of classes and methods. Valid classes are classes that can be searched for in the database.

[0096] It is worth noting that, unlike certain prior art approaches that focus on a single program (e.g., variable assignments), one or more embodiments advantageously build a database of call to class mappings by examining the use of expressions in method calls, or by analyzing a large repository of programs, or a combination thereof. For example, one or more embodiments perform static analysis on a large corpus of code to build a database of method calls for classes, rather than analyzing the contents of a single program using, for example, declarative import statements. In one or more embodiments, a reflection mechanism, or server source code, or a combination thereof, is not required (e.g., because in one or more embodiments, Turtle is a facade that stands in place of the server code). Advantageously, one or more embodiments do not need to execute the code and go through the complexities of dynamic analysis; rather, one or more embodiments perform static analysis on a large number (e.g., millions) of code files to collect a large database of calls based on data flow analysis. This database is then used to infer return types based on the methods invoked on the object being inferred. Indeed, instead of a single source file, one or more embodiments perform static analysis on millions of dynamic code files (e.g., for dynamically typed languages such as PYTHON), thus building concrete mappings between calls and classes based on data analysis. By performing efficient static analysis of a large number (e.g., millions) of programs, one or more embodiments can have an enhanced ability to collect more calls between programs for any particular object type. Thus, one or more embodiments employ duck typing across large code repositories to (a) infer types on a given program with many method calls and extrapolate to others, or (b) infer types using multiple method calls between programs and infer types across a collection of programs with the same API call, or a combination thereof. One or more embodiments provide type inference for dynamically typed languages such as PYTHON.

[0097] One or more embodiments therefore provide techniques for using a computing device to infer types in a dynamic programming language, the techniques including: receiving, by the computing device, a set of programs and a database of all classes (each class including one or more methods); applying, by the computing device, static analysis to each program in the set of programs (assuming that any first instance of a call to any imported library in each program returns a new object of unknown type); using data flow analysis, the computing device identifying one or more subsequent method calls to the new object of unknown type and mapping the results; applying, by the computing device, the mapping to analyze the new program having calls to the imported library; and inferring, by the computing device, a type returned by the call based on the mapping.

[0098] Given the foregoing discussion, it will be appreciated that, in general terms, one aspect of the present invention provides an exemplary method for improving the technical process of programming a computer using a dynamic programming language. The method includes generating a first portion of training data mapping types to corresponding functions and methods in the dynamic programming language by performing an information search on a documentation library in the dynamic programming language. This can be performed, for example, using the techniques described above for type inference using documentation strings, similar to Algorithm 1 in FIG. 5. Suitable commercially available search engines and analysis engines can be employed, as can high-level code implementing type inference / Algorithm 1 compiled or interpreted into computer-executable code.

[0099] Also included is training a neural network on a first portion of training data to infer unknown types in the dynamic programming language. Those skilled in the art are familiar with training computerized neural networks and, given the teachings herein, will be able to train a computerized neural network on the generated data. Also included is inferring unknown types for program variables in a sample of code written in the dynamic programming language by using the trained neural network to perform inference, inferring the unknown types based on the existence of the corresponding functions and methods. Those skilled in the art are familiar with inference using computerized neural networks and, given the teachings herein, will be able to perform inference using a computerized neural network trained on the generated data. See, for example, lines 10-13 of Figure 2. The training data indicates that given types have specific functions and methods. Based on this, inference is made, essentially determining what types the variables are likely to have.

[0100] Further included is a step of facilitating programming in a dynamic programming language based on the inferred unknown types. Various programming activities can be performed based on the inferred unknown types. For example, most dynamic languages, including JavaScript® and PYTHON, include optional syntax for expressing type information. An embodiment of the present invention can generate type information for an existing program, which effectively creates a new program that differs from the original program by adding annotations. That is, in this aspect, programming involves creating a new program based on the existing program, with the new program including the inferred type declarations. Another exemplary programming activity is understanding common things programmers do with a given data set based on abstractions between different programs. This activity is also useful in the fields of artificial intelligence and machine learning, for example. For example, suppose you read a CSV file and delete some columns from the file. Suppose you also perform another operation on the CSV file without dropping any columns. Another program may read the CSV file and perform the same operation directly without dropping any columns. Currently, programmers cannot understand that the two programs are doing the same thing because they cannot understand the type the second method is calling, leading current technology to erroneously infer that the two programs are doing different things. One or more embodiments can be used to obtain a level of abstraction about what code is doing. Another exemplary programming activity is using typing in accordance with embodiments of the present invention to optimize code in a runtime system or compiler. The actual programming can be performed by a human programmer using, for example, a program editor. This activity can be facilitated by making the inferred types available to the programmer in an appropriate file or other data structure.

[0101] In some embodiments, the method further includes performing data flow analysis on a plurality of existing programs written in the dynamic programming language to generate a second portion of training data that maps program variables of the dynamic programming language to corresponding functions and methods of the dynamic programming language, and performing a union operation on the first and second portions of training data to obtain combined training data. This can be performed, for example, using the techniques described above for analytical type inference and an appropriate static analysis software tool. In this embodiment, training the neural network includes training on the combined training data to infer unknown types in the dynamic programming language. TJ Watson Libraries for Analysis (WALA) provides static analysis capabilities for Java bytecode and the related language, JavaScript, and is available from International Business Machines Corporation and may be utilized.

[0102] Furthermore, regarding generating the second part of the training data that maps the above program variables through dataflow analysis, this training data can be generated from multiple existing programs, reducing the number of possible types during inference. For example, run the program and check what functions and methods are called for each variable (e.g., data, starting from line 10 in Figure 2). From there, infer what type it may hold (only types for which methods and functions are actually implemented can hold it). That is, data must have a head method. Consider data.head on line 13—any of the possible types the variable data may have (objects that may be assigned to data) must have one, since the head method has just been called. This limits the types data may actually have. Since there are only a limited number of types that actually have a head method, it helps determine what type data actually is.

[0103] Further included is a step of reducing the number of types searched during the inference based on the mapping of the program variables. In essence, the search space is constrained and classification is performed on a smaller set of types. If a method is invoked on a variable, that variable can only have types that actually implement the method.

[0104] Thus, all embodiments use both information retrieval on well-documented libraries and data flow analysis on multiple existing programs, although it is often advantageous to use both of these aspects together. However, each aspect can yield useful types. The union of training data is a larger set, and combining both aspects generally improves the performance of each. In set theory, the union of sets of collections is the set of all elements contained in the collections. Those skilled in the art will be familiar with finding the union of sets in modern programming languages such as PYTHON. It will be appreciated that Figure 2 relates to both information retrieval on well-documented libraries (Section 103 read_csv Documentation) and data flow analysis on multiple existing programs (Section 101 read_csv). In one or more embodiments, a knowledge graph is used to create a training set of types. Those skilled in the art will be familiar with knowledge graphs themselves and, given the teachings herein, will be able to implement one or more embodiments using that same knowledge graph.

[0105] In view of the immediately preceding discussion, it will be appreciated that, in some cases, a method for improving the technical process of computer programming using a dynamic programming language includes generating training data mapping program variables in the dynamic programming language to corresponding functions and methods in the dynamic programming language, training a neural network with at least the training data to infer unknown types in the dynamic programming language, reducing the number of types searched during inference based on the mapping of program variables, using the trained neural network to perform the inference and infer the unknown types, and facilitating programming in the dynamic programming language based on the inferred unknown types. Any of the additional steps discussed elsewhere herein can be performed in connection with this aspect, as appropriate. For this aspect, results from methods using information retrieval or similar results obtained using other techniques can be used.

[0106] One or more embodiments further include running the program (eg, during a testing process by a developer, or by an end user after release, or a combination thereof).

[0107] In one or more embodiments, performing information retrieval includes, for each given module in the set of modules in the documentation library (see lines 3-13 of Algorithm 1), collecting the functions and methods declared in the given module (see line 4). Then, for each collected method, collecting the corresponding class and obtaining the corresponding documentation string (see lines 5, 6, and 7), and parsing each documentation string into a reconstructed text including the return value of the class (see line 8). Furthermore, for each collected function and method, parsing each corresponding documentation string into a reconstructed text including the return value of the function (see line 8), creating a document for each function and method (see line 9), and indexing each document in the index (see line 10). Furthermore, for every given module of the class (see line 15), we search the index for all documents corresponding to the given module of the class (see line 16), assign each of the documents to a corresponding function or method and type, and obtain a mapping between types and corresponding functions and methods in the dynamic programming language (see lines 17-19).

[0108] Those skilled in the art will understand that a function in PYTHON is a block of code that performs a specific task, should contain its own scope, is called by its name, and that all functions may contain zero arguments (no arguments) or more than one argument. When a function finishes, it may or may not return one or more values. Those skilled in the art will understand that methods in PYTHON are somewhat similar to functions, except that they are associated with objects / classes. Thus, methods in PYTHON are very similar to functions, with two major differences: methods are implicitly used for the object that called them, and methods can access data contained within a class.

[0109] Furthermore, in PYTHON, a type refers to the type of data such as text, number, boolean etc., the type() function returns the class type of a given variable, a class is a code template for creating objects, and an object is a collection of data (variables) and methods / functions that act on the data.

[0110] Appropriate post-treatments such as washing can be carried out as appropriate (see exemplary explanation above).

[0111] In one or more embodiments, generating the first and second portions of training data is unsupervised.

[0112] 21, 22 and the accompanying text, in one or more embodiments, performing the data flow analysis on the plurality of existing programs written in the dynamic programming language includes applying static analysis to each of the programs assuming that a call to an imported library returns a new object of unknown type at 2202, identifying subsequent method calls on the new object of unknown type and determining the union of all methods called on the new object of unknown type at 2203, identifying a corresponding function for each of the calls at 2204, creating a union of all methods in the programs for the corresponding function at 2205, identifying the classes that contain all methods in the union of all methods for the corresponding functions at 2206, and mapping the corresponding functions to the identified classes at 2207. Further, performing the inference using the trained neural network to infer the unknown type at 2208 includes applying the mapping.

[0113] In another aspect, an exemplary computer (see, e.g., FIG. 25) includes a memory 28 embodying computer-executable instructions 40; and at least one processor 16 coupled to the memory and operable by the computer-executable instructions, capable of performing any one, some, or all of the method steps of any of the methods described herein. For example, the at least one processor operates to instantiate a neural network, and / or a search engine and / or static software analysis tool, depending on whether it is performing information searches on a documentation library, or data flow analysis on a plurality of existing programs, or a combination thereof.

[0114] Although this disclosure includes detailed descriptions of cloud computing, it should be understood that implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment now known or later developed.

[0115] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processes, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and released with minimal management effort or interaction with the service provider. The cloud model can include at least five characteristics, at least three service models, and at least four deployment models.

[0116] The characteristics are as follows:

[0117] On-Demand Self-Service: Cloud consumers can unilaterally provision computing capacity, such as server time and network storage, automatically as needed, without requiring human interaction with the service provider.

[0118] Broad network access: Functionality is available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin- or thick-client platforms (eg, cell phones, laptops, and PDAs).

[0119] Resource Pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. There is location independence in that consumers generally have no control or knowledge over the exact location of the provided resources, but it may be possible to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0120] Rapid Elasticity: This capacity can be rapidly and elastically provisioned, sometimes automatically, to quickly scale out, and rapidly released to quickly scale in. To the consumer, the capacity available for provisioning often appears unlimited, and can be purchased in any quantity at any time.

[0121] Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both providers and consumers of utilized services.

[0122] The service model is as follows:

[0123] Software as a Service (SaaS): The consumer is offered the ability to use a provider's applications running on a cloud infrastructure. The applications are accessible from a variety of client devices through a thin-client interface such as a web browser (e.g., web-based email). The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, storage, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0124] Platform as a Service (PaaS): The ability offered to consumers is to deploy applications they create or acquire, written using programming languages and tools supported by the provider, on a cloud infrastructure. The consumer does not manage or control the underlying cloud infrastructure, including the network, servers, operating systems, or storage, but does control the deployed applications and, in some cases, the application hosting environment configuration.

[0125] Infrastructure as a Service (IaaS): The ability offered to consumers is to provision processing, storage, network, and other basic computing resources, on which the consumer can deploy and run any software, which may include operating systems and applications. The consumer does not manage or control the underlying cloud infrastructure, but does have control over the operating systems, storage, deployed applications, and possibly limited control of selected networking components (e.g., host firewalls).

[0126] The deployment model is as follows:

[0127] Private Cloud: Cloud infrastructure operates solely for an organization. A private cloud may be managed by that organization or a third party and may exist on-premise or off-premise.

[0128] Community Cloud: Cloud infrastructure is shared by several organizations to support a specific community with shared concerns (e.g., mission, security requirements, policy and compliance considerations). It may be managed by those organizations or a third party and may exist on-premises or off-premises.

[0129] Public Cloud: Cloud infrastructure is made available to the general public or large industry organizations and is owned by organizations that sell cloud services.

[0130] Hybrid Cloud: A cloud infrastructure is a composite of two or more clouds (private, community, or public) that remain distinct entities but are bound together by standard or proprietary technologies that allow for data and application portability (e.g., cloud bursting for load balancing between clouds).

[0131] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing is an infrastructure that includes a network of interconnected nodes.

[0132] Referring now to FIG. 23, an exemplary cloud computing environment 50 is shown. As shown, the cloud computing environment 50 includes one or more cloud computing nodes 10 with which local computing devices used by cloud users can communicate, such as, for example, a personal digital assistant (PDA) or cellular phone 54A, a desktop computer 54B, a laptop computer 54C, or an automobile computer system 54N, or any combination thereof. The nodes 10 may communicate with each other. The nodes may be physically or virtually grouped (not shown) in one or more networks, such as a private cloud, a community cloud, a public cloud, or a hybrid cloud, or any combination thereof, as described above. This enables the cloud computing environment 50 to provide infrastructure, a platform, or software, or a combination thereof, as a service without requiring cloud users to maintain resources on their local computing devices. It should be understood that the types of computing devices 54A-N shown in FIG. 23 are intended for illustrative purposes only, and that the computing nodes 10 and the cloud computing environment 50 can communicate with any type of computerized device via any type of network or network-addressable connection, or combination thereof (e.g., using a web browser).

[0133] Referring now to Figure 24, a set of functional abstraction layers provided by cloud computing environment 50 (Figure 23) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 24 are intended to be exemplary only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0134] Hardware and software layer 60 includes hardware and software components. Examples of hardware components include mainframe 61, RISC (reduced instruction set computer) architecture-based servers 62, servers 63, blade servers 64, storage devices 65, and networks and networking components 66. In some embodiments, software components include network application server software 67 and database software 68.

[0135] The virtualization layer 70 provides an abstraction layer from which examples of virtual entities such as virtual servers 71, virtual storage devices 72, virtual networks including virtual private networks 73, virtual applications and operating systems 74, and virtual clients 75 can be sourced.

[0136] In one example, management layer 80 may provide the following functions: Resource provisioning 81 provides dynamic procurement of computing and other resources utilized to execute tasks within the cloud computing environment. Metering and pricing 82 provides cost tracking as resources are utilized within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud users and tasks, and protection for data and other resources. User portal 83 provides consumers and system administrators with access to the cloud computing environment. Service level management 84 provides allocation and management of cloud computing resources so that required service levels are met. Service level agreement (SLA) planning and fulfillment 85 provides advance arrangement and procurement of cloud computing resources where future requirements are anticipated according to SLAs.

[0137] Workload tier 90 provides examples of functionality for which a cloud computing environment may be utilized. Examples of workloads and functionality that may be provided from this tier include mapping and navigation 91, software development and lifecycle management 92, virtual classroom instruction delivery 93, data analytics processing 94, transaction processing 95, and at least portions of a system for type inference in dynamic languages 96.

[0138] One or more embodiments of the present invention, or elements thereof, may be implemented in the form of an apparatus including a memory and at least one processor coupled to the memory and operable to perform exemplary method steps. FIG. 25 illustrates a computer system useful in implementing one or more aspects and / or elements of the present invention and may represent a cloud computing node according to embodiments of the present invention. Referring now to FIG. 25, cloud computing node 10 is merely one example of a suitable cloud computing node and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the present invention described herein. Regardless, cloud computing node 10 may implement and / or perform any of the functionality described herein above.

[0139] Cloud computing node 10 includes computer system / server 12 that operates in conjunction with numerous other general-purpose or application-specific computing system environments or configurations. Examples of well-known computing systems, environments, or configurations, or combinations thereof, that may be suitable for use with computer system / server 12 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics devices, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices.

[0140] The computer system / server 12 may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system / server 12 may be practiced in a distributed cloud computing environment where tasks are performed by remote processing devices linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media, including memory storage devices.

[0141] 25, computer system / server 12 in cloud computing node 10 is shown in the form of a general-purpose computing device. Components of computer system / server 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 coupling various system components including system memory 28 to processor 16.

[0142] Bus 18 represents any one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. By way of example and not limitation, such architectures include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0143] Computer system / server 12 typically includes a variety of computer system-readable media, which can be any available media that can be accessed by computer system / server 12 and includes both volatile and nonvolatile media, and removable and non-removable media.

[0144] The system memory 28 may include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 30 or cache memory 32, or a combination thereof. The computer system / server 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, typically referred to as a "hard drive"). Although not shown, a magnetic disk drive may be provided for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive may be provided for reading from or writing to a removable, non-volatile optical disk, such as a CD-ROM, DVD-ROM, or other optical media. In such cases, each may be connected to the bus 18 by one or more data media interfaces. As further depicted and described below, the memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the present invention.

[0145] By way of example and not limitation, programs / utilities 40 including a set (at least one) of program modules 42 may be stored in memory 28, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data, or any combination thereof, may include an implementation of a networking environment. The program modules 42 generally perform the functions and / or methods of embodiments of the present invention as described herein.

[0146] The computer system / server 12 may communicate with one or more external devices 14, such as a keyboard, pointing device, display 24, etc., one or more devices that allow a user to interact with the computer system / server 12, or any device (e.g., network card, modem, etc.) that allows the computer system / server 12 to communicate with one or more other computing devices, or a combination thereof. Such communication may occur via an input / output (I / O) interface 22. Furthermore, the computer system / server 12 may communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), or a public network (e.g., the Internet) or a combination thereof, via a network adapter 20. As depicted, the network adapter 20 communicates with other components of the computer system / server 12 via a bus 18. Although not shown, it should be understood that other hardware and / or software components may be used in conjunction with the computer system / server 12. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0147] Accordingly, one or more embodiments may utilize software running on a general-purpose computer or workstation. Referring to FIG. 25, such an implementation may employ, for example, a processor 16, memory 28, and a display 24 and an input / output interface 22 to an external device 14, such as a keyboard or pointing device. As used herein, the term "processor" is intended to include any processing device, such as a device including a CPU (Central Processing Unit) or other form of processing circuitry, or both. Furthermore, the term "processor" may refer to two or more individual processors. The term "memory" is intended to include memory associated with a processor or CPU, such as, for example, RAM (Random Access Memory) 30, ROM (Read-Only Memory), fixed memory devices (e.g., hard drives 34), removable memory devices (e.g., diskettes), flash memory, etc. Additionally, as used herein, the expression "input / output interface" is intended to contemplate an interface to, for example, one or more mechanisms for inputting data to a processing unit (e.g., a mouse) and one or more mechanisms for providing results associated with the processing unit (e.g., a printer). The processor 16, memory 28, and input / output interface 22 may be interconnected, for example, via a bus 18 as part of the data processing unit 12. Suitable interconnections, for example, via the bus 18, may also be provided to a network interface 20, such as a network card, which may be provided to interface with a computer network, and to a media interface, such as a diskette or CD-ROM drive, which may be provided to interface with suitable media.

[0148] Thus, computer software containing instructions or code for carrying out the methods of the present invention, as described herein, can be stored in one or more of the associated memory devices (e.g., ROM, fixed or removable memory) and, when ready for use, can be loaded partially or wholly (e.g., into RAM) and implemented by a CPU. Such software can include, but is not limited to, firmware, resident software, microcode, etc.

[0149] A data processing system suitable for storing and / or executing program code includes at least one processor 16 coupled directly or indirectly to memory elements 28 via a system bus 18. The memory elements may include local memory utilized during the actual implementation of the program code, bulk storage, and cache memory 32 that provides temporary storage of at least some program code to reduce the number of times the code must be retrieved from bulk storage during implementation.

[0150] Input / output or I / O devices (including but not limited to keyboards, displays, pointing devices, etc.) can be coupled to the system either directly or through intermediary I / O controllers.

[0151] A network adapter 20 may also be coupled to the system to enable the data processing system to couple to other data processing systems or remote printers or storage devices through intermediary private or public networks. Currently available types of network adapters include modems, cable modems, and Ethernet cards.

[0152] As used herein, including in the claims, a "server" includes a physical data processing system (e.g., system 12 as shown in FIG. 25) running a server program. It should be understood that such a physical server may or may not include a display and keyboard.

[0153] One or more embodiments may be implemented at least in part in the context of a cloud or virtual machine environment, by way of example and not limitation. See again Figures 23-24 and accompanying text.

[0154] It should be noted that any of the methods described herein may include the additional step of providing a system including separate software modules embodied in a computer-readable storage medium, where the modules may include any or all of the appropriate elements, e.g., shown in block diagrams and / or described herein, including, by way of example and not limitation, any one, some, or all of the described modules / blocks, or sub-modules / sub-blocks, or combinations thereof. In this regard, the method steps may be performed using separate software modules and / or sub-modules of such a system executed on one or more hardware processors, such as 16. Furthermore, a computer program product may include a computer-readable storage medium having code adapted to be implemented to perform one or more method steps described herein, including providing a system having separate software modules.

[0155] One example of a user interface that may be employed in some instances is Hypertext Markup Language (HTML) code served by a server or the like to a browser on a user's computing device, which is parsed by the browser on the user's computing device to create a graphical user interface (GUI).

[0156] [Example System and Product Details]

[0157] The present invention may be a system, a method, or a computer program product, or a combination thereof, at any possible level of technical detail integration. The computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions for causing a processor to perform aspects of the present invention.

[0158] A computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, punch cards, or mechanically encoded devices such as ridge structures in grooves having recorded instructions, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transitory signal per se, such as an electric wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse passing through a fiber optic cable), or an electrical signal transmitted through a wire.

[0159] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or may be downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof. The network may include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium within the respective computing / processing device.

[0160] The computer-readable program instructions for carrying out the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for an integrated circuit, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk® or C++, and procedural programming languages such as the “C” programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, to carry out aspects of the present invention, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the electronic circuitry.

[0161] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0162] These computer-readable program instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executed by the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts or block diagrams, or a combination thereof. These computer-readable program instructions may also be stored on a computer-readable storage medium and can instruct a computer, programmable data processing apparatus, or other device, or a combination thereof, to function in a particular manner, such that a computer-readable storage medium having instructions stored thereon includes a product containing instructions that implement an aspect of the function / act specified in a block or blocks of the flowcharts or block diagrams, or a combination thereof.

[0163] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer, other programmable apparatus, or other device implement the function(s) / act(s) specified in one or more blocks of the flowchart(s) or block diagram(s), or a combination thereof.

[0164] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially simultaneously, depending on the functionality involved, or the blocks may even be executed in the reverse order. It should also be noted that each block of a block diagram or flowchart diagram, or a combination thereof, and combinations of blocks in block diagrams or flowchart diagrams, or a combination thereof, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0165] The description of various embodiments of the present invention has been presented for purposes of illustration, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein have been selected to best explain the principles of the embodiments, practical applications, or technical improvements over commercially available technologies, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A method for improving a technical process of computer programming using a dynamic programming language, said method comprising: generating a first portion of training data mapping types in the dynamic programming language to corresponding functions and methods in the dynamic programming language by performing an information lookup on a documentation library in the dynamic programming language; training a neural network on a first portion of the training data to infer unknown types in the dynamic programming language; performing inference using the trained neural network to infer the unknown types of program variables in a sample of code written in the dynamic programming language by inferring the unknown types based on the existence of the corresponding functions and methods; facilitating programming in the dynamic programming language based on the inferred unknown type; A method comprising:

2. generating a second portion of the training data mapping the program variables in the dynamic programming language to the corresponding functions and methods in the dynamic programming language by performing a data flow analysis on a plurality of existing programs written in the dynamic programming language; performing a union operation on the first and second portions of the training data to obtain combined training data; Furthermore, The step of training the neural network includes training the combined training data to infer the unknown types in the dynamic programming language, and further reducing the number of types searched during the inference based on the mapping of program variables. The method of claim 1.

3. The method of claim 2 further comprising the step of executing the program.

4. For each given module of the set of modules in the documentation library, performing the information search includes: collecting the functions and methods declared in each given module, For each of the collected methods, collecting the corresponding classes and obtaining the corresponding documentation strings; parsing each of the corresponding documentation strings into a reconstructed text containing the return value of the class; and For each of the collected functions and methods: parsing each of the corresponding documentation strings into a reconstructed text that further includes a function return value; creating documentation for each of said functions and said methods; indexing an index for each of said documents; and For every given module of said class: searching the index for all of the documents corresponding to a given module of the class; assigning each of the documents to a corresponding function or method and a type to obtain the mapping between the types and the corresponding functions and methods in the dynamic programming language; to carry out, stage have The method of claim 2.

5. performing the data flow analysis on the plurality of existing programs written in the dynamic programming language, applying static analysis to each of said programs assuming that calls to imported libraries will return new objects of unknown types; identifying subsequent method calls to new objects of said unknown type; determining the union of all methods invoked on the new object of the unknown type; identifying a corresponding function for each of said calls; creating a union of all methods of said program with respect to said corresponding functions; identifying a class that includes all methods in the union of all methods for the corresponding functions; mapping the corresponding functions to the identified classes; and performing the inference using the trained neural network and inferring the unknown type comprises applying the mapping. The method of claim 2.

6. The method of claim 2 , wherein generating the first and second portions of training data is unsupervised.

7. A computer program embodying computer-executable instructions which, when executed by a computer, cause the computer to perform a method for improving the technical process of programming the computer using a dynamic programming language, the method comprising: generating a first portion of training data mapping types in the dynamic programming language to corresponding functions and methods in the dynamic programming language by searching a documentation library in the dynamic programming language; training a neural network on a first portion of the training data to infer unknown types in the dynamic programming language; performing inference using the trained neural network to infer the unknown types of program variables in a sample of code written in the dynamic programming language by inferring the unknown types based on the existence of the corresponding functions and methods; facilitating programming in the dynamic programming language based on the inferred unknown type; have Computer program.

8. The method implemented by the computer executing the computer-executable instructions comprises: generating a second portion of the training data mapping the program variables in the dynamic programming language to the corresponding functions and methods in the dynamic programming language by performing a data flow analysis on a plurality of existing programs written in the dynamic programming language; performing a union operation on the first and second portions of the training data to obtain combined training data; and The step of training the neural network includes training the combined training data to infer the unknown types in the dynamic programming language, and further reducing the number of types searched during the inference based on the mapping of program variables.

8. A computer program according to claim 7.

9. 9. The computer program product of claim 8, wherein the computer-implemented method of executing the computer-executable instructions further comprises the step of executing the program product.

10. For each given module of the set of modules in the documentation library, the step of performing the information search comprises: a procedure for collecting the functions and methods declared in each given module, the procedure comprising: For each of the collected methods, A procedure for collecting the corresponding classes and retrieving the corresponding documentation strings; parsing each of the corresponding documentation strings into a reconstructed text that includes the return value of the class; and For each of the collected functions and methods: parsing each of the corresponding documentation strings into a reconstructed text that further includes a return value of the function; creating documentation for each of said functions and methods; indexing each of said documents into an index; and For all of the given modules of a class, searching the index for all of the documents corresponding to a given module of the class; assigning each of the documents to a corresponding function or method and a type to obtain the mapping between the types and the corresponding functions and methods in the dynamic programming language; To do this, follow these steps: Contains 9. A computer program according to claim 8.

11. The step of performing the data flow analysis on the plurality of existing programs written in the dynamic programming language comprises: applying static analysis to each of said programs assuming that calls to imported libraries will return new objects of unknown types; identifying subsequent method calls to the new object of the unknown type; determining the union of all methods invoked on the new object of said unknown type; identifying a corresponding function for each of said calls; creating a union of all the methods of said program with respect to said corresponding functions; identifying a class that includes all methods in the union of all methods for the corresponding functions; mapping the corresponding functions to the identified classes; Including, performing the inference using the trained neural network to infer the unknown type includes applying the mapping; 9. A computer program according to claim 8.

12. 12. The computer program of claim 8, wherein the steps of generating the first and second portions of training data are unsupervised.

13. a memory embodying computer-executable instructions; at least one processor, coupled to the memory and operated by the computer-executable instructions, for performing a method for improving a technological process of programming a computer using a dynamic programming language, The method comprises: instantiating a neural network and a search engine; generating a first portion of training data mapping types in the dynamic programming language to corresponding functions and methods in the dynamic programming language by searching for information about a documentation library in the dynamic programming language using at least the search engine; training a neural network on a first portion of the training data to infer unknown types in the dynamic programming language; performing inference using the trained neural network to infer the unknown types of program variables in a sample of code written in the dynamic programming language by inferring the unknown types based on the existence of the corresponding functions and methods; facilitating programming in the dynamic programming language based on the inferred unknown type; have at least one processor; A computer comprising:

14. The at least one processor is further operated by the computer-executable instructions; Steps for instantiating a static software analysis tool; generating a second portion of the training data mapping the program variables in the dynamic programming language to the corresponding functions and methods in the dynamic programming language by performing a data flow analysis on a plurality of existing programs written in the dynamic programming language using at least the static software analysis tool; performing a union operation on the first and second portions of the training data to obtain combined training data; Execute The step of training the neural network includes training the combined training data to infer the unknown types in the dynamic programming language, and further reducing the number of types searched during the inference based on the mapping of program variables. The computer of claim 13.

15. The computer of claim 14 , wherein the at least one processor is operated by the computer-executable instructions to further cause the program to execute.

16. For each given module of the set of modules in the documentation library, the step of performing the information search comprises: a procedure for collecting the functions and methods declared in each given module, the procedure comprising: For each of the collected methods, A procedure for collecting the corresponding classes and retrieving the corresponding documentation strings; parsing each of the corresponding documentation strings into a reconstructed text containing the return value of the class; and For each of the collected functions and methods: parsing each of the corresponding documentation strings into a reconstructed text that further includes a return value of the function; creating documentation for each of said functions and methods; indexing each of said documents into an index; and For every given module of said class: searching the index for all of the documents corresponding to a given module of the class; assigning each of the documents to a corresponding function or method and a type to obtain the mapping between the types and the corresponding functions and methods in the dynamic programming language; To do this, follow these steps: Contains The computer of claim 14.

17. The step of performing the data flow analysis on the plurality of existing programs written in the dynamic programming language comprises: applying said static software analysis tool to each of said programs, assuming that calls to imported libraries will return new objects of unknown types; identifying subsequent method calls to the new object of the unknown type; determining the union of all methods invoked on the new object of said unknown type; identifying a corresponding function for each of said calls; creating a union of all the methods of said program with respect to said corresponding functions; identifying a class that includes all methods in the union of all methods for the corresponding functions; mapping the corresponding functions to the identified classes; Including, performing the inference using the trained neural network to infer the unknown type includes applying the mapping; The computer of claim 14.

18. 18. The computer of claim 14, wherein the procedure for generating the first and second portions of training data is unsupervised.

19. 1. A method for improving a technical process of computer programming using a dynamic programming language, said method comprising: generating training data that maps program variables in the dynamic programming language to corresponding functions and methods in the dynamic programming language by performing data flow analysis on a plurality of existing programs written in the dynamic programming language; training a neural network with at least the training data to infer unknown types in the dynamic programming language; reducing the number of types searched during inference based on the mapping of program variables; performing the inference using the trained neural network to infer the unknown type; facilitating programming in the dynamic programming language based on the inferred unknown type; A method comprising:

20. performing the data flow analysis on the plurality of existing programs written in the dynamic programming language, applying static analysis to each of said programs assuming that calls to imported libraries will return new objects of unknown types; identifying subsequent method calls to new objects of said unknown type; determining the union of all methods invoked on the new object of the unknown type; identifying a corresponding function for each of said calls; creating a union of all methods of said program with respect to said corresponding functions; identifying a class that includes all methods in the union of all methods for the corresponding functions; mapping the corresponding functions to the identified classes; and performing the inference using the trained neural network to infer the unknown type comprises applying the mapping.

20. The method of claim 19.

Images