API recommendation method based on natural language query

Abstract

The invention particularly relates to an API recommendation method based on natural language query. The API recommendation method based on natural language query comprises the following steps: crawling a Java open source project on a code hosting platform Githhub by using a Scrapy crawler framework, and obtaining information of each storage library; selecting an API library as a target library in a user-defined manner, analyzing and extracting the source code by using a code analysis tool, and obtaining API method calling information of the target library; s2, randomly selecting metadata from all the metadata obtained in the step S2 as a data set, performing clustering analysis on the data set by using a hierarchical clustering algorithm, and mining an API use mode from the data set; and performing API recommendation according to the API use mode. According to the API recommendation method based on natural language query, analysis and mode mining are performed on large-scale codes, API recommendation can be performed according to the input natural language text described by the function, the accuracy of recommending related APIs is improved, the workload of searching and selecting the APIs by developers is reduced, and the efficiency of software development is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of software engineering, and in particular to an API recommendation method based on natural language query. Background Art

[0002] API is widely used in modern software development. Calling API is one of the important ways for developers to quickly build software and improve development efficiency. With the development of the software industry and the expansion of software scale, software engineering has become more and more complex, and the number of third-party libraries has also increased day by day. They provide a large number of APIs that can be called to help developers simplify the development process. However, there are also shortcomings. For these numerous third-party APIs, it is almost impossible for developers to fully understand them, and it is difficult to make choices based on their own experience. Due to the uneven quality of API documents, it takes a lot of time to write documents, and it is difficult to maintain them in real time. It is easy to have problems such as document lag, inaccurate descriptions, and ambiguity. In addition, the functional requirements of developers are often abstract and extensive, while the functional descriptions of APIs in API documents are usually more specific. It also takes a lot of time to consult the documents, which causes obstacles for developers in the process of using API documents.

[0003] In order to solve this problem, relevant recommendation technology is needed to recommend API usage patterns (i.e. a set of API calls required to implement a certain function) to developers. Existing recommendation schemes often take a certain API or code context as input and recommend a series of related APIs. With only a vague concept of the required API function, it is difficult to determine the specific API or code.

[0004] In order to reduce the workload of developers in searching and selecting APIs and improve the efficiency of software development, the present invention proposes an API recommendation method based on natural language query. Summary of the invention

[0005] In order to overcome the deficiencies of the prior art, the present invention provides a simple and efficient API recommendation method based on natural language query.

[0006] The present invention is achieved through the following technical solutions:

[0007] A method for recommending an API based on natural language query, characterized in that it comprises the following steps:

[0008] Step S1: Crawl Java open source projects

[0009] Use the Scrapy crawler framework to crawl Java open source projects on the code hosting platform Github and obtain information from each repository;

[0010] Step S2: Extract metadata

[0011] Customize and select the API library as the target library, use code analysis tools to analyze and extract the source code, and obtain the API method call information of the target library;

[0012] Step S3: Mining API usage patterns

[0013] Randomly select metadata from all metadata obtained in step S2 as a data set, perform cluster analysis on the data set using a hierarchical clustering algorithm, and mine API usage patterns therefrom;

[0014] Step S4: API recommendation

[0015] Make API recommendations based on API usage patterns.

[0016] In step S1, the implementation steps are as follows:

[0017] Step S101: Use the code hosting platform GitHub to construct and obtain the query address of the data API on the code hosting platform Github, and use it as the initial link for the crawler to crawl;

[0018] Step S102, query and search Java projects with star numbers between 100 and 100,000, sort by star number in descending order, 100 items per page, and return results in Json format;

[0019] Step S103: parse the returned results to obtain information of each repository in the search results.

[0020] In step S2, based on the usage frequency of third-party API libraries and referring to the target libraries selected in previous similar studies, 9 API libraries including commons-io, commons-codec, commons-lang, commons-logging, easymock, log4j, slf4j, servlet-api and junit are finally selected as target libraries.

[0021] In step S2, the API method call information of the target library is extracted from the Java open source project based on the AST module in Eclipse JDT; the specific steps are as follows:

[0022] Step S201: for a MethodDeclaration type node, determine whether there is comment information:

[0023] If it does not exist, it returns false and stops traversing its child nodes;

[0024] If it exists, extract the comment information and method name, return true, and continue to traverse the child nodes;

[0025] Step S202: when traversing to the MethodInvocation subnode under the MethodDeclaration type node, obtain API method call information, ignore method calls within the project, and only retain third-party API method calls;

[0026] Step S203, traverse all files in all open source projects, and for Java source code files with the suffix .java, parse them into abstract syntax trees through ASTParser, and then use the custom abstract syntax tree visitor ASTVisitor to traverse and visit the abstract syntax tree to obtain method names, API calls and comment information in the code files.

[0027] In step S3, the implementation steps are as follows:

[0028] Step S301: First, define a similarity measurement rule, and define the similarity between metadata by combining method names and API calls;

[0029] Step S302: Edit distance, i.e. Levenshtein distance, is used to quantify the difference between two character strings.

[0030] Define the minimum similarity between two metadata belonging to two classes as the inter-class similarity between the two classes, and merge the two classes with the largest inter-class similarity until the inter-class similarity between all classes is lower than the custom threshold;

[0031] In step S302, the custom threshold is set to 0.5. When the inter-class similarity of all classes is lower than 0.5, it is considered that all similar metadata have been clustered in one class, and there is no class that meets the merging condition, so clustering is stopped.

[0032] Step S303 : hierarchically clustering the metadata using a hierarchical clustering algorithm to obtain a number of clusters, each of which is a collection of a large number of similar metadata, and each cluster is a different API usage mode.

[0033] In step S4, the implementation steps are as follows:

[0034] Step S401: taking the annotation set of the clusters obtained by hierarchical clustering as the description information of the API usage pattern, and converting the description information of the API usage pattern into a weight vector in a vector space model using a TF-IDF weight calculation method;

[0035] Step S402, pre-processing the input natural language query text of the function description and converting it into a weight vector in a vector space model;

[0036] Step S403, respectively calculating the cosine similarity between the natural language query text and each API usage pattern, sorting the calculated cosine similarities in descending order from high to low, and finally outputting the API usage pattern with the highest similarity as the recommendation result.

[0037] A system for implementing the above-mentioned API recommendation method based on natural language query, comprising:

[0038] The Java open source project crawling module is responsible for crawling Java open source projects on the code hosting platform Github using the Scrapy crawler framework to obtain information from each repository;

[0039] The metadata extraction module is responsible for customizing the selection of the API library as the target library, using the code analysis tool to analyze and extract the source code, and obtaining the API method call information of the target library;

[0040] The API usage pattern mining module is responsible for randomly selecting metadata from all metadata obtained by the metadata extraction module as a data set, and using a hierarchical clustering algorithm to perform cluster analysis on the data set to mine API usage patterns;

[0041] The API recommendation module is responsible for making API recommendations based on API usage patterns.

[0042] The beneficial effects of the present invention are: the API recommendation method based on natural language query, by analyzing and mining patterns of large-scale codes, can make API recommendations based on the natural language text of the input function description, thereby improving the accuracy of recommending related APIs, reducing the workload of developers in finding and selecting APIs, and improving the efficiency of software development. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Attached Figure 1 This is a schematic diagram of the API recommendation method based on natural language query of the present invention. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0046] Due to the continuous development of software libraries, APIs are updated and iterated very quickly. APIs may become obsolete, and there is competition between third-party APIs with similar functions. Developers need to consider functions, convenience, stability, security, etc. before choosing third-party APIs. How to obtain and analyze the code data of a large number of Java projects on Github and use this data to implement an API recommendation system is a research problem with practical significance.

[0047] The API recommendation method based on natural language query includes the following steps:

[0048] Step S1: Crawl Java open source projects

[0049] Use the Scrapy crawler framework to crawl Java open source projects on the code hosting platform Github and obtain information from each repository;

[0050] Step S2: Extract metadata

[0051] Customize and select the API library as the target library, use code analysis tools to analyze and extract the source code, and obtain the API method call information of the target library;

[0052] Step S3: Mining API usage patterns

[0053] Randomly select metadata from all metadata obtained in step S2 as a data set, perform cluster analysis on the data set using a hierarchical clustering algorithm, and mine API usage patterns therefrom;

[0054] Step S4: API recommendation

[0055] Make API recommendations based on API usage patterns.

[0056] In step S1, the implementation steps are as follows:

[0057] Step S101: Use the code hosting platform GitHub to construct and obtain the query address of the data API on the code hosting platform Github, and use it as the initial link for the crawler to crawl; the example is as follows:

[0058] https: / / api.github.com / search / repositories?q=stars:100..100000+language:Java&sort=stars&order=desc&per_page=100&page=1

[0059] Step S102, query and search Java projects with star numbers between 100 and 100,000, sort by star number in descending order, 100 items per page, and return results in Json format;

[0060] Step S103: parse the returned results to obtain information of each repository in the search results.

[0061] In order to test the effect of using this method, the source code (about 251GB) of all 15,969 Java open source projects with more than 100 stars is obtained according to step S1. 75,500 metadata are randomly selected from all the metadata obtained as the data set. The hierarchical clustering algorithm is used to perform cluster analysis on the data set, and the API usage pattern is mined from it as shown in the table:

[0062] Table 1 Statistics of API usage patterns

[0063] API Library Number of API usage patterns commons-io 440 commons-codec 116 commons-lang 1129 commons-logging 371 easymock 89 log4j 532 slf4j 186 junit 520 servlet-api 1611 total 4994

[0064] When the developer wants to delete a file directory, he enters delete a directory and the output is as follows:

[0065] org.apache.commons.io.FileUtils#cleanDirectory

[0066] org.apache.commons.io.FileUtils#deleteDirectory

[0067] org.apache.commons.io.FileUtils#forceDelete

[0068] In step S2, based on the usage frequency of third-party API libraries and referring to the target libraries selected in previous similar studies, 9 API libraries including commons-io, commons-codec, commons-lang, commons-logging, easymock, log4j, slf4j, servlet-api and junit are finally selected as target libraries.

[0069] In step S2, the API method call information of the target library is extracted from the Java open source project based on the AST module in Eclipse JDT; the specific steps are as follows:

[0070] Step S201: for a MethodDeclaration type node, determine whether there is comment information:

[0071] If it does not exist, it returns false and stops traversing its child nodes;

[0072] If it exists, extract the comment information and method name, return true, and continue to traverse the child nodes;

[0073] Step S202: when traversing to the MethodInvocation subnode under the MethodDeclaration type node, obtain API method call information, ignore method calls within the project, and only retain third-party API method calls;

[0074] Step S203, traverse all files in all open source projects, and for Java source code files with the suffix .java, parse them into abstract syntax trees through ASTParser, and then use the custom abstract syntax tree visitor ASTVisitor to traverse and visit the abstract syntax tree to obtain method names, API calls and comment information in the code files.

[0075] In step S3, the implementation steps are as follows:

[0076] Step S301: First, define a similarity measurement rule, and define the similarity between metadata by combining method names and API calls;

[0077] Step S302: Edit distance, i.e. Levenshtein distance, is used to quantify the difference between two character strings.

[0078] Define the minimum similarity between two metadata belonging to two classes as the inter-class similarity between the two classes, and merge the two classes with the largest inter-class similarity until the inter-class similarity between all classes is lower than the custom threshold;

[0079] In step S302, the custom threshold is set to 0.5. When the inter-class similarity of all classes is lower than 0.5, it is considered that all similar metadata have been clustered in one class, and there is no class that meets the merging condition, so clustering is stopped.

[0080] Step S303 : hierarchically clustering the metadata using a hierarchical clustering algorithm to obtain a number of clusters, each of which is a collection of a large number of similar metadata, and each cluster is a different API usage mode.

[0081] In step S4, the implementation steps are as follows:

[0082] Step S401: taking the annotation set of the clusters obtained by hierarchical clustering as the description information of the API usage pattern, and converting the description information of the API usage pattern into a weight vector in a vector space model using a TF-IDF weight calculation method;

[0083] Step S402, pre-processing the input natural language query text of the function description and converting it into a weight vector in a vector space model;

[0084] Step S403, respectively calculating the cosine similarity between the natural language query text and each API usage pattern, sorting the calculated cosine similarities in descending order from high to low, and finally outputting the API usage pattern with the highest similarity as the recommendation result.

[0085] The system for implementing the API recommendation method based on natural language query includes:

[0086] The Java open source project crawling module is responsible for crawling Java open source projects on the code hosting platform Github using the Scrapy crawler framework to obtain information from each repository;

[0087] The metadata extraction module is responsible for customizing the selection of the API library as the target library, using the code analysis tool to analyze and extract the source code, and obtaining the API method call information of the target library;

[0088] The API usage pattern mining module is responsible for randomly selecting metadata from all metadata obtained by the metadata extraction module as a data set, and using a hierarchical clustering algorithm to perform cluster analysis on the data set to mine API usage patterns;

[0089] The API recommendation module is responsible for making API recommendations based on API usage patterns.

[0090] In summary, the API recommendation method based on natural language query uses the Scrapy crawler framework to crawl Java open source projects on Github, uses code analysis tools to analyze and extract source code, puts related API methods in the top ranking position for recommendation, and uses hierarchical clustering to merge similar methods in the dataset, such as overloaded methods, that is, methods with the same method name but different parameter lists. API recommendations are made based on the natural language text of the input function description, which improves the accuracy of recommending related APIs, reduces the workload of developers in finding and selecting APIs, and improves the efficiency of software development.

[0091] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. An API recommendation method based on natural language query, characterized by: The following steps are involved: Step S1: Crawl Java open source projects Use the Scrapy crawler framework to crawl Java open source projects on the code hosting platform Github and obtain information from each repository; Step S2: Extract metadata Customize and select the API library as the target library, use code analysis tools to analyze and extract the source code, and obtain the API method call information of the target library; Step S3: Mining API usage patterns Randomly select metadata from all metadata obtained in step S2 as a data set, perform cluster analysis on the data set using a hierarchical clustering algorithm, and mine API usage patterns therefrom; Step S4: API recommendation Make API recommendations based on API usage patterns.

2. The API recommendation method based on natural language query according to claim 1, characterized in that: In step S1, the implementation steps are as follows: Step S101: Use the code hosting platform GitHub to construct and obtain the query address of the data API on the code hosting platform Github, and use it as the initial link for the crawler to crawl; Step S102, query and search Java projects with star numbers between 100 and 100,000, sort by star number in descending order, 100 items per page, and return results in Json format; Step S103: parse the returned results to obtain information of each repository in the search results.

3. The API recommendation method based on natural language query according to claim 1, characterized in that: In step S2, nine API libraries including commons-io, commons-codec, commons-lang, commons-logging, easymock, log4j, slf4j, servlet-api and junit are selected as target libraries.

4. The API recommendation method based on natural language query according to claim 1, characterized in that: In step S2, the API method call information of the target library is extracted from the Java open source project based on the AST module in Eclipse JDT; the specific steps are as follows: Step S201: for a MethodDeclaration type node, determine whether there is comment information: If it does not exist, return false and stop traversing its child nodes; If it exists, extract the comment information and method name, return true, and continue to traverse the child nodes; Step S202: when traversing to the MethodInvocation subnode under the MethodDeclaration type node, obtain API method call information, ignore method calls within the project, and only retain third-party API method calls; Step S203, traverse all files in all open source projects, and for Java source code files with the suffix .java, parse them into abstract syntax trees through ASTParser, and then use the custom abstract syntax tree visitor ASTVisitor to traverse and visit the abstract syntax tree to obtain method names, API calls and comment information in the code files.

5. The API recommendation method based on natural language query according to claim 1, characterized in that: In step S3, the implementation steps are as follows: Step S301: First, define a similarity measurement rule, and define the similarity between metadata by combining method names and API calls; Step S302: Edit distance, i.e. Levenshtein distance, is used to quantify the difference between two character strings. Define the minimum similarity between two metadata belonging to two classes as the inter-class similarity between the two classes, and merge the two classes with the largest inter-class similarity until the inter-class similarity between all classes is lower than the custom threshold; Step S303: hierarchical clustering algorithm is used to perform hierarchical clustering on the metadata to obtain a number of clusters, each of which is a different API usage mode.

6. The API recommendation method based on natural language query according to claim 5, characterized in that: In step S302, the custom threshold is set to 0.

5. When the inter-class similarity of all classes is lower than 0.5, it is considered that all similar metadata have been clustered in one class, and there is no class that meets the merging condition, so clustering is stopped.

7. The API recommendation method based on natural language query according to claim 5, characterized in that: In step S4, the implementation steps are as follows: Step S401: taking the annotation set of the clusters obtained by hierarchical clustering as the description information of the API usage pattern, and converting the description information of the API usage pattern into a weight vector in a vector space model using a TF-IDF weight calculation method; Step S402, pre-processing the input natural language query text of the function description and converting it into a weight vector in a vector space model; Step S403, respectively calculating the cosine similarity between the natural language query text and each API usage pattern, sorting the calculated cosine similarities in descending order from high to low, and finally outputting the API usage pattern with the highest similarity as the recommendation result.

8. A system for implementing the API recommendation method based on natural language query according to any one of claims 1 to 7, characterized in that: The Java open source project crawling module is responsible for crawling Java open source projects on the code hosting platform Github using the Scrapy crawler framework to obtain information from each repository; The metadata extraction module is responsible for customizing the selection of the API library as the target library, using the code analysis tool to analyze and extract the source code, and obtaining the API method call information of the target library; The API usage pattern mining module is responsible for randomly selecting metadata from all metadata obtained by the metadata extraction module as a data set, and using a hierarchical clustering algorithm to perform cluster analysis on the data set to mine API usage patterns; The API recommendation module is responsible for making API recommendations based on API usage patterns.

Images