Retrieval method for marking program codes based on metadata keywords

Abstract

The invention discloses a retrieval method for marking program codes based on metadata keywords, which comprises the following steps of: crawling project source codes based on a git or svn command interface in combination with a self-defined crawler program, splitting the project source codes into proper code segments, storing the code segments into a search engine, and establishing related indexes; finding out a user additional mark comment or a program code segment, database table data and structure information containing related input contents according to user input and in combination with related indexes in a search engine; by recording frequently-used search values of users, analyzing records and regularly counting frequently-used search keywords and hot words, indexes are reconstructed according to analysis results, and sorting of retrieval results is optimized. The method has the advantages that the marking annotation, comment and program code retrieval efficiency is greatly improved to the second level, meanwhile, a retrieval result list sequence is optimized and perfected in combination with user-defined keyword marking metadata, and efficient and convenient support is provided for code retrieval, replication and reuse.

Description

Technical Field

[0001] The present invention relates to the technical field of program source code management, and in particular to a retrieval method for program code based on metadata keyword marking. Background Art

[0002] Program source code management in the field of information management has always been one of the core tasks of information technology companies; the program source code of the information management system is one of the core assets of information technology companies, and is the crystallization of the wisdom, knowledge, and hard work of all the staff who created the information management system of information technology companies. How to reuse the program source code of the information management system more efficiently, cherish the fruits of labor, save labor, save energy for information technology companies, and reduce costs and increase efficiency for the construction of information management systems has always been an important direction for information technology companies to work hard for their own companies.

[0003] While information technology has made great progress and accumulated fruitful results, a large number of established information management systems have overlapping business implementations and inconsistent processing flow logic, forming information barriers and information islands. One of the reasons for these problems is insufficient program modularization and code reuse. How to make program code reuse more efficient? One of the solutions is to enhance program readability and expand program modules from code-level reuse. However, faced with the massive program source code accumulated by information technology companies and the time cycle of new information management system projects, it is impossible for staff who plan to reuse to read through all the program codes in the historical program source code library. It is time-consuming and laborious to search for project codes one by one, and the search methods are limited. It is not easy to add comments and annotations (see the relationship diagram). Figure 1 ). Therefore, how to assist information technology enterprises to quickly retrieve and present cross-project codes, improve the reuse rate of historical project codes, provide references for the development of similar or identical functions for new projects, and efficiently retrieve program codes across projects is a problem that needs to be solved at present. Summary of the invention

[0004] The object of the present invention is to provide a retrieval method for program code based on metadata keyword tagging, thereby solving the above-mentioned problems existing in the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for retrieving program codes based on metadata keyword tags comprises the following steps:

[0007] S1. Source code crawling and indexing:

[0008] Crawl the project source code based on git or svn command interface combined with custom crawler program, split the project source code into appropriate code segments, store them in the search engine and create relevant indexes;

[0009] S2. Program code retrieval:

[0010] Based on the user input and the relevant index in the search engine, find the user's additional markup comments or program code segments, database table data and structure information containing the relevant input content;

[0011] S3. Optimize the sorting of search results:

[0012] By recording users' frequently used search values, analyzing the records and regularly counting commonly used search keywords and hot words, the index is reconstructed based on the analysis results to optimize the search result sorting.

[0013] Preferably, step S1 specifically includes the following contents:

[0014] S11. Apply for database and table permissions, write crawlers, crawl database table data, and establish data and database, table relationship indexes;

[0015] S12. Apply for program code permissions, write a crawler, crawl the program source code and parse the comment part and store it separately, reserve comment-related fields, and establish an index of source code path, project, source code, comments, and comment-related information.

[0016] Preferably, in step S1, for a program capable of parsing and separating program source code, a program source code and comment parsing and separation operation is performed, and by adding a parser plug-in to the code crawler program, the data organization format in the database table and its corresponding storage index field relationship are allowed to be reorganized.

[0017] Preferably, the java program source code uses the javaparservi s ited library for parsing and comment separation; the C#.net program source code uses Microsoft's CSharpSyntaxVi s itor library for parsing and comment separation; the css program source code uses the ph-css library for parsing and comment separation; the html or html program source code uses jsoup for parsing and comment separation; and the javascript program source code uses estree for parsing and comment separation.

[0018] Preferably, step S4 includes two methods for optimizing the ranking of search results:

[0019] A. Count the frequently searched values ​​of users every week, analyze and record the frequently searched keywords and hot words, and adjust the click rate of matching hot word records based on the analysis results, so that the results matching the hot words in the past week can be displayed first when displaying search results based on the click rate;

[0020] B. Re-analyze user comments and annotations based on the analysis and statistics of the hot words of the week and search for keywords, and rebuild the index. Increase the index weight of the matching hot words by a preset multiple, so that the results matching the hot words of the past week can be displayed first when displaying search results based on the click-through rate.

[0021] Preferably, code retrieval is implemented based on a code retrieval interface, which is open to third-party calls and access, supports third-party plug-ins for retrieving code, and returns relevant program code fragments or all according to appropriate permissions or protocols for reuse or reuse reference.

[0022] Preferably, based on the code retrieval and result presentation interface, the search content is input and the search results are displayed. The interface highlights the code of the relevant program language according to the program syntax, and provides a convenient code copy button and code source file download button. The code copy button can be used to copy the code segment with one click, and the code source file download button can be used to download the program source file with one click. The associated user comments and marks are displayed on the side of the interface.

[0023] Preferably, the custom crawler program is a crawler program written for the database table structure and its data, and the project program source code.

[0024] Preferably, a custom crawler program is executed regularly to dynamically pull source code of each project based on a git or svn command interface, update the indexed program code, add the original markup comments in the code, and retain the tags inherited from the old code in the same path.

[0025] Preferably, before step S1, it also includes environment building configuration, and the environment building configuration specifically includes the following contents:

[0026] S01. Download and configure OpenSerach;

[0027] S02. Download and install git / svn and configure the code pulling environment;

[0028] S03. Install openJDK.

[0029] The beneficial effects of the present invention are: 1. It enables program code reusers to quickly find program codes that may need to be reused in a large amount of historical codes, and reuse, expand or rewrite them for reference, thereby improving work efficiency in the newly built information management system, reducing their own work costs, and thus reducing the cost of building and using the information management system and improving its efficiency. 2. It can avoid the time-consuming and laborious traditional project-based code dynamic search and retrieval through IDE (integrated development environment) (usually takes several hours or days, and does not support additional markup comments), greatly improve the efficiency of markup annotations, comments and program code retrieval to seconds, and at the same time combine user-defined keyword markup metadata to optimize and improve the order of the search result list, providing efficient and convenient support for code retrieval, replication and reuse. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a relationship diagram for code management and data / code search in traditional information enterprises;

[0031] Figure 2 is an information enterprise code management relationship diagram based on the method of the present invention in an embodiment of the present invention;

[0032] Figure 3 is an index entity relationship diagram in an embodiment of the present invention;

[0033] Figure 4 Schematic diagram of crawling tasks and management engine in an embodiment of the present invention ( Figure 2 );

[0034] Figure 5 It is a roadmap for optimizing and improving index retrieval results based on user keyword tags, annotations and comments in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0036] like Figure 2 As shown, in this embodiment, a retrieval method based on metadata keyword marking program code is provided, including the following steps:

[0037] 1. Source code crawling and indexing

[0038] Based on the git or svn command interface combined with a custom crawler program, the project source code is crawled, the project source code is split into appropriate code segments, stored in the search engine and related indexes are established. Specifically, the following contents are included:

[0039] (1) Apply for database and table permissions, write crawlers, crawl database table data, and establish indexes on the relationship between data, databases, and tables;

[0040] (2) Apply for program code permissions, write a crawler, crawl the program source code, parse the comment part and store it separately, reserve comment-related fields, and establish an index of source code path, project, source code, comments, and comment-related information.

[0041] The above two steps are mainly scheduled crawler programs, just like Internet crawlers, which dynamically pull the source code of each project based on git / svn commands, update the program code that has been indexed, add the original markup comments in the code, and retain the tags inherited from the old code in the same path. Figure 3 .

[0042] The crawler program regularly executes the process of "pulling program source files, reading, parsing and writing scheduled tasks into search engine indexes".

[0043] In this embodiment, by adding a parser plug-in to the code crawler program, the data organization format in the mapping database table and its corresponding storage index field relationship can be reorganized to achieve separation of program source code parsing and annotation (custom label / marker set).

[0044] It is not necessary to separate program code from comments, but it is something that needs to be done as much as possible. For Java program source files, use the javaparser library to parse and separate the program code and all the comment statements in the code; C#.net program source files use Microsoft's CSharpSyntaxVisitor library to parse the code and separate all the code and comments in the code source file; css program source files use the ph-css library to parse; html or htm source files use jsoup to parse; javascript program source files use estree to parse. Programs in other languages ​​that cannot parse and separate program source code do not need to be separated. All program languages ​​support users to add comments and annotations to the search results, and the contents of these added comments and annotations can be retrieved and matched first.

[0045] In this embodiment, for a program that is compiled and executed, the program source code refers to the compilable code portion within the program source file, as opposed to the comment code within the program source file that will be ignored during compilation; for a program that is interpreted and executed, the program source code refers to the code portion that will not be ignored during execution, as opposed to the comment portion within the program source file that will be ignored during execution.

[0046] In this embodiment, Figure 4As shown, a custom crawler program is executed regularly to dynamically pull the source code of each project based on the git or svn command interface, and the indexed program code is regularly established or updated according to the configured tasks, and the original mark comments in the code are added, while retaining the tags inherited from the old code in the same path.

[0047] 2. Program code retrieval

[0048] Based on the user input and the relevant index in the search engine, find the user's additional markup comments or program code segments, database table data and structure information containing the relevant input content.

[0049] In this embodiment, an open code retrieval interface is provided, which supports third-party plug-ins for retrieving codes and returns relevant program code fragments or all of them according to appropriate permissions or protocols for reuse or reuse reference.

[0050] It also provides a code search and result presentation interface, which highlights the code of the relevant programming language according to the program syntax, a convenient copy button to copy code segments with one click, a one-click download source code file button to download program source files, and displays related comments and tags on the side.

[0051] 3. Optimize the sorting of search results

[0052] By recording the commonly used search values ​​of users, analyzing the records and regularly counting the commonly used search keywords and hot words, the index is reconstructed based on the analysis results to optimize the search result sorting. Figure 5 As shown, there are two specific methods:

[0053] A. Count the users' frequently searched values ​​every week, analyze and record the frequently searched keywords and hot words, and adjust the click rate of matching hot word records based on the analysis results, so that the results matching the hot words in the past week can be displayed first when displaying search results based on the click rate.

[0054] B. Re-analyze user comments and annotations based on the analysis and statistics of the weekly hot words and search for keywords, and rebuild the index. Increase the index weight of the matching hot words by a preset multiple (such as 2 times), so that the results matching the hot words of the past week can be displayed first when displaying search results based on click-through rate.

[0055] Among them, click-through rate is a built-in ranking indicator field when the OpenSearch search engine creates an index.

[0056] In this embodiment, the retrieval method requires environment setup and configuration before implementation, specifically including:

[0057] (1) Download and configure OpenSerach;

[0058] (2) Download and install git / svn and configure the code pulling environment;

[0059] (3) Install openJDK.

[0060] In this embodiment, a search test can be performed every day to verify the accuracy of the search result matching and the response efficiency. The index segmentation is optimized based on the hot words that users often search. By adjusting the index structure, increasing or optimizing the server configuration, the response time can be reduced and the response efficiency can be improved.

[0061] In this embodiment, the retrieval method is implemented based on the retrieval system. The system includes a search engine based on OpenSearch, a customized implementation of the project source code and database table structure, and a data crawler program that stores data into the OpenSearch search engine and creates an index, implements a general code retrieval and result presentation interface and an open code retrieval interface, an interface for adding tags and comments to the retrieval results, and optimizes the order of retrieval results based on user usage and marked comments.

[0062] The OpenSearch search engine stores and provides interface support for rapid retrieval of program code database table index data. Through customization, multiple project source codes and database table structures of information technology companies and their data crawler programs are crawled and stored in the search engine OpenSearch and a full-text search index is established. The system provides a code search and result presentation interface and opens a code search service interface. It adds a tag comment form to the search result interface and optimizes the order of user search results based on user search matching. A convenient code copy button and code source file download button are provided on the code search result interface to provide quick operation function buttons for program reuse. Through weighted matching of programs marked and commented by users, comments in the project program code are matched first, and the codes matching user searches are sorted according to the weighted comprehensive click-through rate.

[0063] Search engine: It is a very mature full-text search technology. Based on this concept, there are some mature open source software projects, such as OpenSearch used in this method; OpenSearch is a search engine redeveloped by Amazon based on the ElasticSearch 7 branch and licensed under the Apache 2.0 agreement. It supports independent or clustered (on-demand) deployment, word segmentation, full-text index storage, and efficient full-text search. This search engine program suite can be selected as the core program code index storage and search engine in the implementation.

[0064] Customized crawler program: Since the invention method is to improve the rapid retrieval and search of the core asset project code of the information technology enterprise, the crawler program prepared for the search engine to retrieve data in this method is different from the crawler program in the usual sense. It is a crawler program written for the database table structure and its data, and the project program source code. And for the project program source code, it is parsed and split according to the method as much as possible and stored in association.

[0065] Code retrieval and result presentation interface: Similar to traditional Internet search engines, it provides a unified search input box and its result display interface.

[0066] Open code search interface: While the search interface is provided for code search and result presentation, it is also open to third-party calls and supports RESTful calls.

[0067] Interface for adding tags and comments to search results: In the interface that supports quick search and result presentation, a quick input box and submit button form are added for commenting on search results and adding custom tag annotations.

[0068] Optimize search result sorting: prioritize results that match user tags, comments, and annotations.

[0069] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0070] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A method for searching program codes based on metadata keyword tags, characterized in that: The following steps are included: S1. Source code crawling and indexing: Crawl the project source code based on git or svn command interface combined with custom crawler program, split the project source code into appropriate code segments, store them in the search engine and create relevant indexes; S2. Program code retrieval: Based on the user input and the relevant index in the search engine, find the user's additional markup comments or program code segments, database table data and structure information containing the relevant input content; S3. Optimize the sorting of search results: By recording users' frequently used search values, analyzing the records and regularly counting commonly used search keywords and hot words, the index is reconstructed based on the analysis results to optimize the search result sorting.

2. The method for retrieving program code based on metadata keyword tagging according to claim 1, characterized in that: Step S1 specifically includes the following contents: S11. Apply for database and table permissions, write crawlers, crawl database table data, and establish data and database, table relationship indexes; S12. Apply for program code permissions, write a crawler, crawl the program source code and parse the comment part and store it separately, reserve comment-related fields, and establish an index of source code path, project, source code, comments, and comment-related information.

3. The method for retrieving program code based on metadata keyword tagging according to claim 2, characterized in that: In step S1, for programs that can parse and separate program source code, the program source code and comment parsing and separation operation is performed, and by adding a parser plug-in to the code crawler program, the data organization format in the database table and its corresponding storage index field relationship can be reorganized.

4. The method for retrieving program code based on metadata keyword tagging according to claim 1, characterized in that: The java program source code uses the javaparservisited library for parsing and comment separation; the C#.net program source code uses Microsoft's CSharpSyntaxVisitor library for parsing and comment separation; the css program source code uses the ph-css library for parsing and comment separation; the html or htm program source code uses jsoup for parsing and comment separation; the javascript program source code uses estree for parsing and comment separation.

5. The method for retrieving program code based on metadata keyword tagging according to claim 1, characterized in that: Step S4 includes two methods for optimizing the ranking of search results: A. Count the frequently searched values ​​of users every week, analyze and record the frequently searched keywords and hot words, and adjust the click rate of matching hot word records based on the analysis results, so that the results matching the hot words in the past week can be displayed first when displaying search results based on the click rate; B. Re-analyze user comments and annotations based on the analysis and statistics of the hot words of the week and search for keywords, and rebuild the index. Increase the index weight of the matching hot words by a preset multiple, so that the results matching the hot words of the past week can be displayed first when displaying search results based on the click-through rate.

6. The method for retrieving program code based on metadata keyword tagging according to claim 1, characterized in that: Code retrieval is implemented based on a code retrieval interface, which is open to third-party calls and access, supports third-party plug-ins for code retrieval, and returns relevant program code fragments or all according to appropriate permissions or protocols for reuse or reuse reference.

7. The method for retrieving program code based on metadata keyword tagging according to claim 6, characterized in that: Based on the code search and result presentation interface, the search content is input and the search results are displayed. The interface highlights the code of the relevant program language according to the program syntax, and provides a convenient code copy button and code source file download button. The code copy button can be used to copy the code segment with one click, and the code source file download button can be used to download the program source file with one click. The related user comments and marks are displayed on the side of the interface.

8. The method for retrieving program code based on metadata keyword tagging according to claim 1, characterized in that: The custom crawler program is a crawler program written for the database table structure and its data, and the project program source code.

9. The method for retrieving program code based on metadata keyword tagging according to claim 1, characterized in that: Execute custom crawler programs regularly, dynamically pull source code of each project based on git or svn command interface, update the indexed program code, add the original mark comments in the code, and retain the tags inherited from the old code in the same path.

10. The method for retrieving program codes based on metadata keyword marking according to any one of claims 1 to 9, characterized in that: Before step S1, the environment building configuration is also included, and the environment building configuration specifically includes the following contents: S01. Download and configure OpenSerach; S02. Download and install git / svn and configure the code pulling environment; S03. Install openJDK.

Images