Visual retrieval method for open source project

Abstract

The invention discloses a visual retrieval method for an open source project. The method comprises the steps of extracting key information of the open source project, analyzing a dependency relationship to extract dependency item information, classifying README files, extracting functional images and availability indexes, and training a BERTopic model to extract theme information. Establishing an index, performing hierarchical topic modeling through topic information, and retrieving and screening items in combination with a semantic sequence; key identifiers such as file names and function names are extracted by analyzing the source code file, and the occurrence frequency of the key identifiers is counted; based on the correlation and similarity of the items, generating an expanded item recommendation list; and performing multi-dimensional display on a retrieval result through a visualization technology. According to the visualization process, data are displayed by adopting multiple visual angles, and dynamic interaction between a user and a search result is realized through an interactive operation interface, so that the user is supported to deeply explore and analyze the search result.

Description

Technical Field

[0001] The present invention relates to the field of computer software, and in particular to a visual retrieval method for open source projects. Background Art

[0002] Searching and reusing existing projects in the open source community can help developers improve programming efficiency. However, with the rapid development of computer technology, the scale of open source projects has increased dramatically, and the information of open source projects (such as project descriptions and dependent libraries) has become increasingly rich. According to statistics from GitHub, the largest open source software community, as of September 2024, GitHub has more than 182 million registered users and more than 198 million public code repositories. Due to the existence of massive open source project information, the lack of an effective retrieval mechanism may cause developers to spend a lot of time and energy in finding suitable projects. The task of efficiently retrieving open source projects according to user needs needs to be solved urgently.

[0003] GitHub provides a text-based search engine that uses keywords as input and returns relevant projects based on the match between the keywords and the project title or description, rather than matching and retrieving based on the specific functions of the project. Although GitHub supports advanced search, it only provides simple multiple filter combinations such as project creation date and number of stars. Users still need to understand the large number of returned projects by reading documents and source code, which is a time-consuming and tedious task. As a result, users usually only focus on browsing the first three pages of content, thus missing many potentially relevant projects.

[0004] Existing research has applied traditional machine learning methods to improve retrieval accuracy, such as generating item tags or classifying items. However, these tools are unable to generate appropriate query keywords, which may result in missing relevant items. In addition, many tools use collaborative filtering (CF) technology to recommend items by analyzing the preferences of similar users. For example, user-based collaborative filtering is used to generate item tags, or item-based collaborative filtering is used to recommend items. However, these methods suffer from the cold start problem. Although researchers have proposed various methods to improve retrieval accuracy, users still need to perform a tedious item inspection process. There are also several problems that need to be addressed, especially: (1) Low transparency of results. Users can only access item information (such as stars, item description, and owner) of the current page, rather than all retrieval results. (2) Information is scattered. Users must explore each item and identify the relationship between items by checking their links. (3) The burden of understanding is heavy. Users need to spend a lot of time locating files in projects with deep file hierarchies and numerous files. Summary of the invention

[0005] In view of the above problems existing in the prior art, the purpose of the present invention is to provide a visual retrieval method for open source projects to improve the accuracy of retrieval results.

[0006] In order to achieve the above object, the present invention provides a visual retrieval method for open source projects, and the specific steps are as follows:

[0007] Step 1: Get several open source projects from the open source community:

[0008] Extract key information for each open source project, including the project name, author information, number of observers, number of stars, number of branches, project description, project size, functional image, license information, contributor information, programming language and its number of code lines, as well as the project's page link and source code link;

[0009] Step 2: Resolve and organize library dependencies:

[0010] From each open source project, parse the corresponding package.json file and lib / libs folder to extract the project's dependency information, including the dependency name, dependencies and their hierarchical structure. By dividing the prefix of the dependency name, the hierarchical structure with parent-child node relationship is extracted. The extracted dependencies and their hierarchical structure will be written into the predefined {techs} field for subsequent index establishment;

[0011] Step 3. Extract and categorize the README file:

[0012] The README file provided by each open source project is used to describe the functions of the open source project. The multi-label classifier READMEClassifier is used to classify the content of each README file into eight categories, namely What, Why, How, When, Who, References, Contribution, and Other. The text classified into the What and Why categories is extracted as the functional text;

[0013] Step 4. Extract functional images and add usability indicators:

[0014] Use the BeautifulSoup library to extract functional images from each open source project, and the links of the extracted functional images will be written into the predefined {images} field; the availability of the open source project is defined as whether there is a How category in the README file. If there is a How category, add a new field {hasHow:True}; otherwise, add {hasHow:False}. Both the {images} and {hasHow} fields will be used for subsequent indexing;

[0015] Step 5: Extract topics based on project information BERTopic

[0016] The key information of each project is merged to obtain the merged information, and the BERTopic model is trained based on the merged information to extract the topic, topic number, corresponding topic keywords and keyword weights corresponding to the keywords of each open source project. The extracted topic keywords will be recorded in the predefined {topics} field for subsequent index establishment and hierarchical topic modeling.

[0017] Step 6: Create an index

[0018] All the information extracted in steps 1 to 5 above is uniformly input into Elasticsearch, and an index is created for each open source project.

[0019] Step 7, hierarchical topic modeling: Open source projects with the same topic are classified into the same topic class, and then the topic classes are clustered using a clustering method. Two topic classes with a distance less than a preset threshold are classified into one cluster. When the topic classes in each cluster no longer change, clustering is completed.

[0020] Step 8. Match the query statement with the project:

[0021] Elasticsearch is used to search the query statement entered by the user on each cluster clustered in step 7. The returned results include all the hit open source projects, the fields in the hit open source projects that match the query statement, and the relevance scores corresponding to the matched fields. Each field includes the field name and its corresponding field value, and the content in the field value that matches the query term is highlighted.

[0022] For each hit open source project, the field with the highest relevance score among all matching fields is selected as the optimal field, the query statement entered by the user is decomposed into several independent query terms, and the highlighted content in the optimal field is matched with several query terms one by one to obtain the semantic order of the optimal field.

[0023] Step 9: Parse the source code of all hit open source projects:

[0024] For the open source projects that are hit, Esprima is first used to parse the source code files and generate the corresponding abstract syntax tree AST.

[0025] Step 10: Recommend open source projects that are related and similar to the hit open source project:

[0026] All hit open source projects are taken as a recommendation list, and other open source projects that are relevant and similar to the hit open source projects are added to the recommendation list to obtain a final recommendation list.

[0027] Step 11: Visualize all the results from step 1 to step 10.

[0028] Furthermore, the step of obtaining the optimal semantic order of the fields in step 8 includes:

[0029] Step 8-1: Label all query terms in the query statement from left to right, and label the highlighted content corresponding to the query term in the optimal field with the same label as the query term.

[0030] Step 8-2: The optimal field includes N sentences, N≥1, and the labels of all query terms in each sentence are formed into an array, and repeated labels in the array are deleted.

[0031] Step 8-3: Connect the N arrays at the end to obtain a one-dimensional array composed of labels. When the elements in the one-dimensional array are repeated periodically from left to right, keep the elements in the first cycle and delete the elements in other cycles to obtain the semantic order of the optimal fields in the query statement in the open source projects hit.

[0032] Furthermore, in step 10, the step of obtaining the final recommendation list includes:

[0033] Step 10-1: Use elasticsearch to find other open source projects of the owner or contributor of the hit open source project; if the number of lines of code contributed by each contributor to the hit open source project exceeds the set threshold, the contributor will be included in the contributor search list; if the number of lines contributed by the contributor is lower than this threshold, the first three contributors will be included in the contributor search list by default. Also use elasticsearch to search for other open source projects of the contributors included in the list; define the other open source projects of the owner or contributor found as other related open source projects.

[0034] Step 10-2: The project name, description, and README text of each project in other related open source projects are processed and merged into a string list. The index of the string list represents the number of each project, and the list element represents the key information of each project.

[0035] The same method is used to process and merge the project name, description, and README text of each hit open source project into a string list.

[0036] Step 10-3: Calculate the cosine similarity between the hit open source project and other related open source projects, sort them in descending order according to the cosine similarity value, add the top ten related open source projects to the recommendation list, and obtain the final recommendation list.

[0037] Furthermore, in step 11, the visual display includes five components: a basic interface component, a theme overview view component, a theme clustering view component, a project details view component and a global control panel.

[0038] Basic interface components: including three sub-components: search bar, result filtering and sorting, and view switching. The search bar is used to enter keywords or query statements to start the search operation; result filtering and sorting are used to select and display "all" or "part" of the open source projects in a sub-node in the subject tree, and sort the open source projects according to the set indicators; view switching is used to automatically expand and collapse the selected open source projects;

[0039] The topic overview view component is used to display the topic tree of all open source projects in the final recommended project list. The topic overview view component also includes a control module for adjusting parameters.

[0040] The topic clustering view component is used to display the clustering results in a chart and also to display the usage frequency of the open source projects in the clustering results.

[0041] The project details view component consists of three parts: basic project information, source code analysis result visualization, and project recommendation table. Basic project information is used to display the frequency of use of folder names, file names, function names, and variable names. Visualization of source code analysis results is used to display the source code of all hit open source projects. The project recommendation table is used to display the final recommendation column.

[0042] Global dashboard component: used to display the popularity index, project update time, and matching order of open source projects in the final recommendation column.

[0043] Compared with the prior art, the present invention has at least the following advantages:

[0044] 1. BERTopic extracts topics and keywords from texts such as project descriptions and README files based on semantic understanding. Compared with traditional machine learning methods, BERTopic can not only divide projects into different topics, but also provide keywords for each topic and the corresponding keyword weights. This effectively solves the problem that traditional tools cannot generate suitable query keywords, and reduces the risk of users missing relevant projects due to keyword mismatches.

[0045] 2. Through BERTopic topic extraction technology, the scattered project information is aggregated into clear topic classes based on semantic relevance, and the topic classes are further arranged from broad to specific through hierarchical modeling, constructing a multi-level structure that progresses from superordinate topics to subordinate subtopics. This method effectively solves the problem of scattered information and avoids the tedious operation of users exploring projects and their links one by one to identify the relationship between projects during the retrieval process. The hierarchical display of semantic associations improves the transparency of retrieval results, provides a more efficient and intuitive way of presenting results, significantly improves the efficiency of users' information acquisition, and optimizes the overall user experience.

[0046] 3. Compare the order of terms in the query statement with the matching order of these terms in the project field with the highest relevance score, and further filter out relevant projects that meet the specified matching order, thereby effectively solving the technical problem that users must explore each project one by one to identify the relationship between projects due to scattered information.

[0047] 4. Esprima parses the source code and generates an abstract syntax tree (AST), accurately extracts key identifiers such as folders, files, functions, and variable names, and counts their frequencies. Based on these identifiers and their frequency information, a word cloud is generated. Users can click on the identifiers in the word cloud to view the file location and further access the source code. This method effectively solves the difficulty of users locating target files in complex file hierarchies, significantly reduces the burden of understanding, and improves retrieval efficiency.

[0048] 5. The present invention generates a list of recommended projects by calculating the relevance and similarity of projects, providing users with more projects that may be of interest, effectively solving the problem of information overload caused by the large number of open source projects, and helping users quickly find related projects that meet their needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A diagram showing a system of the present invention.

[0050] Figure 2 To group the experimental results, a performance comparison chart of their average values ​​is presented using a bar chart and a line chart.

[0051] Figure 3 System architecture diagram of the present invention.

[0052] Figure 4 The process of finding the optimal semantic order of fields. DETAILED DESCRIPTION

[0053] The present invention is further described in detail below

[0054] The present invention provides a visual retrieval method for open source projects, and the specific steps are as follows:

[0055] Step 1: Get several open source projects from the open source community:

[0056] Extract key information for each open source project, including the project name, author information, number of observers, number of stars, number of branches, project description, project size, functional image, license information, contributor information, programming language and its number of code lines, as well as the project's page link and source code link;

[0057] Step 2: Resolve and organize library dependencies:

[0058] From each open source project, parse the corresponding package.json file and lib / libs folder to extract the project's dependency information, including the dependency name, dependencies and their hierarchical structure. By dividing the prefix of the dependency name, the hierarchical structure with parent-child node relationship is extracted. The extracted dependencies and their hierarchical structure will be written into the predefined {techs} field for subsequent index establishment;

[0059] The library dependencies reflect the way the project is developed. Dependencies are extracted from the package.json file and the lib / libs folder. To reduce data redundancy, the version numbers in the extracted library names are removed. Dependency names are divided by prefixes to extract dependencies with parent-child node relationships;

[0060] Step 3. Extract and categorize the README file:

[0061] The README file provided by each open source project is used to describe the functions of the open source project. The multi-label classifier READMEClassifier is used to classify the content of each README file into eight categories, namely What, Why, How, When, Who, References, Contribution, and Other. The text classified into the What and Why categories is extracted as the functional text;

[0062] Step 4. Extract functional images and add usability indicators:

[0063] The images in the functional text usually show the functions and execution results of the project, so the BeautifulSoup library is used to extract the functional images of each open source project, and the links of the extracted functional images will be written into the predefined {images} field; the availability of the open source project is defined as whether there is a How category in the README file. If there is a How category, it means that the project has high availability, and a new field {hasHow:True} is added; otherwise, {hasHow:False} is added. Both the {images} and {hasHow} fields will be used for subsequent index establishment;

[0064] Step 5: Extract topics based on project information BERTopic

[0065] The key information of each project is merged to obtain the merged information, and the BERTopic model is trained based on the merged information to extract the topic, topic number, corresponding topic keywords and keyword weights corresponding to the keywords of each open source project. The extracted topic keywords will be recorded in the predefined {topics} field for subsequent index establishment and hierarchical topic modeling.

[0066] Step 6: Create an index

[0067] All the information extracted in steps 1 to 5 above is uniformly input into Elasticsearch, and an index is created for each open source project.

[0068] Step 7, hierarchical topic modeling: Open source projects with the same topic are classified into the same topic class, and then the topic classes are clustered using a clustering method. Two topic classes with a distance less than a preset threshold are classified into one cluster. When the topic classes in each cluster no longer change, clustering is completed.

[0069] Since the generated topics may be related, a hierarchical relationship structure between all topics is further constructed based on the subject word matrix. The construction of this hierarchical relationship is based on calculating the distance between each topic, such as using a measurement method such as cosine similarity. First, starting from each independent topic, the two most similar topics are merged into a new cluster in turn, and then the topics with the highest similarity between clusters are continuously merged until all topics are gradually integrated into the same hierarchical structure. The merging process of the above topics can be achieved through a variety of linkage functions, including but not limited to single linkage, complete linkage or average linkage. Using the generated hierarchical relationship structure of the topic, a relationship tree is constructed for the display of the topic clustering view in step 11. By organizing the topics in a hierarchical manner, the semantic relationship between the topics can be more intuitively displayed. This structure not only helps users quickly identify the core topics and their subtopics related to the keywords, but also makes the association of complex information clearer and more explicit. In this way, users can filter projects more efficiently from a semantic level, improving the transparency and accuracy of search results.

[0070] Step 8. Match the query statement with the project:

[0071] Elasticsearch is used to search the query statement entered by the user on each cluster clustered in step 7. The returned results include all the hit open source projects, the fields in the hit open source projects that match the query statement, and the relevance scores corresponding to the matched fields. Each field includes the field name and its corresponding field value, and the content in the field value that matches the query term is highlighted.

[0072] For each hit open source project, the field with the highest relevance score among all matching fields is selected as the optimal field. Since different arrangement orders of query terms may convey different semantic information, the query statement entered by the user is decomposed into several independent query terms, and the highlighted content in the optimal field is matched with several query terms one by one to obtain the semantic order of the optimal field.

[0073] Step 9: Parse the source code of all hit open source projects:

[0074] For the hit open source projects, Esprima is first used to parse the source code file and generate the corresponding abstract syntax tree AST. By traversing the abstract syntax tree, multiple key identifiers including folder names, file names, function names, and variable names can be extracted, and the frequency of occurrence of these identifiers is counted. In addition, the method further includes calculating the similarity between these key identifiers to achieve the merging of some key identifiers. However, in view of the complexity and time-consuming nature of the similarity calculation process, the merging operation is not performed by default. Based on the parsed key identifiers and their corresponding frequency information, a word cloud is generated for the display of the project details view in step 11. The generated word cloud effectively helps users quickly identify key identifiers and focus on high-frequency content, so that users can intuitively understand the important information of the retrieved project, significantly improving the efficiency of project understanding and analysis.

[0075] Step 10: Recommend open source projects that are related and similar to the hit open source project:

[0076] All hit open source projects are taken as a recommendation list, and other open source projects that are relevant and similar to the hit open source projects are added to the recommendation list to obtain a final recommendation list.

[0077] The recommendation list is used to display the project details view in step 11, providing users with more projects that may be of interest.

[0078] Step 11: Visualize all the results from step 1 to step 10.

[0079] Visualize the retrieved data from multiple angles: Use visualization technology to display project data, including but not limited to basic information, subject matter, and source code of the project. This visualization process uses multiple perspectives to display data, and uses an interactive operation interface to enable dynamic interaction between users and search results, to support users' in-depth exploration and analysis of search results.

[0080] Furthermore, the step of obtaining the optimal semantic order of the fields in step 8 includes:

[0081] Step 8-1: Label all query terms in the query statement from left to right, and label the highlighted content corresponding to the query term in the optimal field with the same label as the query term.

[0082] Step 8-2: The optimal field includes N sentences, N≥1, and the labels of all query terms in each sentence are formed into an array, and repeated labels in the array are deleted.

[0083] Step 8-3: Connect the N arrays at the end to obtain a one-dimensional array composed of labels. When the elements in the one-dimensional array are repeated periodically from left to right, keep the elements in the first cycle and delete the elements in other cycles to obtain the semantic order of the optimal fields in the query statement in the open source projects hit.

[0084] This will further optimize the relevance assessment of the retrieved items and significantly improve the accuracy of the retrieval results.

[0085] For example, the query is machine learning AI, and the three query words are labeled 0, 1, and 2 respectively. If the optimal field of a hit open source project is readme, which includes 4 sentences, the highlighted field is as follows: Figure 4 . Use curIndex = query_arr.index(curItem) to find the index position of the word in the query word array (query_arr = ["machine", "learning", "AI"]).

[0086] Assume that the query term array is: query_arr = ["machine", "learning", "AI"], then:

[0087] curItem="machine" curIndex="0

[0088] curItem="learning" curIndex=1

[0089] curItem="AI"curIndex="2"

[0090] For a hit open source project, the optimal field has 4 sentences. The labels of all query words in each sentence are used to form an array, that is, four arrays are obtained, namely [0,1], [2,2][0,1,2], [0,1,2]. After removing the repeated labels in each array, we get: [0,1], [2][0,1,2], [0,1,2]. The four arrays are connected to get a one-dimensional array: [0,1,2,0,1,2,0,1,2]. From left to right, periodic repetitions begin from the four elements. The repeated second cycle [0,1,2] and the third cycle [0,1,2] are deleted, and only the first cycle [0,1,2] is retained.

[0091] Furthermore, in step 10, the step of obtaining the final recommendation list includes:

[0092] Step 10-1: Use elasticsearch to find other open source projects of the owner or contributor of the hit open source project; if the number of lines of code contributed by each contributor to the hit open source project exceeds the set threshold, the contributor will be included in the contributor search list; if the number of lines contributed by the contributor is lower than this threshold, the first three contributors will be included in the contributor search list by default. Also use elasticsearch to search for other open source projects of the contributors included in the list; define the other open source projects of the owner or contributor found as other related open source projects.

[0093] Step 10-2: Process the project name, description, and README text of each project in other related open source projects [the processing here includes word segmentation, stop word removal, abbreviation replacement, stemming, and part-of-speech restoration] and merge them into a string list. The index of the string list represents the number of each project, and the list elements represent the key information of each project.

[0094] The same method is used to process and merge the project name, description, and README text of each hit open source project into a string list.

[0095] Step 10-3: Calculate the cosine similarity between the hit open source project and other related open source projects, sort them in descending order according to the cosine similarity value, add the top ten related open source projects to the recommendation list, and obtain the final recommendation list.

[0096] Furthermore, in step 11, the visual display includes five components: a basic interface component, a theme overview view component, a theme clustering view component, a project details view component and a global control panel.

[0097] Basic interface components: including three sub-components: search bar, result filtering and sorting, and view switching. The search bar is used to enter keywords or query statements to start the search operation; result filtering and sorting are used to select and display "all" or "part" of the open source projects in a sub-node in the subject tree, and sort the open source projects according to the set indicators; view switching is used to automatically expand and collapse the selected open source projects;

[0098] The topic overview view component is used to display the topic tree of all open source projects in the final recommended project list. The topic overview view component also includes a control module for adjusting parameters.

[0099] The topic clustering view component is used to display the clustering results in a chart and also to display the usage frequency of the open source projects in the clustering results.

[0100] The project details view component consists of three parts: basic project information, source code analysis result visualization, and project recommendation table. Basic project information is used to display the frequency of use of folder names, file names, function names, and variable names. Visualization of source code analysis results is used to display the source code of all hit open source projects. The project recommendation table is used to display the final recommendation column.

[0101] Global dashboard component: used to display the popularity index, project update time, and matching order of the open source projects in the final recommendation column.

[0102] Embodiment 1: This embodiment discloses a visual search method for open source projects, which specifically includes the following steps:

[0103] Step 1: Get open source projects from the open source community:

[0104] Extract key information for each open source project, including the project name, author information, number of observers, number of stars, number of branches, project description, project size, functional image, license information, contributor information, programming language and number of lines of code, as well as the project's page link and source code link. For example, this instance selected GitHub as the data source, cloned JavaScript projects created between January 2010 and December 2023, totaling 74,103 projects, to extract key information.

[0105] Step 2: Parse and organize library dependencies:

[0106] From each open source project, parse the corresponding package.json file and lib / libs folder to extract the project's dependency information. By dividing the prefix of the dependency name, extract the hierarchical structure with a parent-child node relationship. For example, "@eslint / plugin-react-hooks" and "@eslint-plugin-react-router" have the same prefix @eslint, so they are both children of the @eslint node. The extracted dependencies and their hierarchical structure will be written to the predefined {techs} field for subsequent indexing.

[0107] Step 3. Extract and categorize the README file:

[0108] The README file provided by each open source project is used to describe the functions of the open source project. The multi-label classifier READMEClassifier is used to classify the content of each README file into eight categories, namely What, Why, How, When, Who, References, Contribution, and Other. The text classified into the What and Why categories is extracted as the functional text;

[0109] Step 4. Extract functional images and add usability indicators:

[0110] The images in the functional text usually show the functions and execution results of the project, so the links of the functional images of each open source project are extracted using the BeautifulSoup library, and the extracted image links will be written into the predefined {images} field; the availability of the open source project is defined as whether there is a How category in the README file. If there is a How category, it means that the project has high availability, and a new field {hasHow:True} is added; otherwise, {hasHow:False} is added. Both the {images} and {hasHow} fields will be used for subsequent indexing.

[0111] Step 5: Extract topics based on project information BERTopic

[0112] The key information of each project (including project name, description, README text) is merged and processed, and the BERTopic model is trained based on the merged information to extract the topic of each project. The topics extracted through this process will be recorded in the predefined {topics} field for subsequent indexing and hierarchical topic modeling.

[0113] Step 6: Create an index

[0114] All the information extracted in steps 1 to 5 above is uniformly input into Elasticsearch and an index is created for each project.

[0115] Step 7, hierarchical topic modeling: Open source projects with the same topic are classified into the same topic class, and then the topic classes are clustered using a clustering method. Two topic classes with a distance less than a preset threshold are classified into one cluster. When the topic classes in each cluster no longer change, clustering is completed.

[0116] Since the generated topics may be related, a hierarchical relationship structure between all topics is further constructed based on the subject word matrix. The construction of this hierarchical relationship is based on calculating the distance between each topic, such as using a measurement method such as cosine similarity. First, starting from each independent topic, the two most similar topics are merged into a new cluster in turn, and then the topics with the highest similarity between clusters are continuously merged until all topics are gradually integrated into the same hierarchical structure. The merging process of the above topics can be achieved through a variety of linkage functions, including but not limited to single linkage, complete linkage or average linkage. Using the generated hierarchical relationship structure of the topic, a relationship tree is constructed for the display of the topic clustering view in step 11. By organizing the topics in a hierarchical manner, the semantic relationship between the topics can be more intuitively displayed. This structure not only helps users quickly identify the core topics and their subtopics related to the keywords, but also makes the association of complex information clearer and more explicit. In this way, users can filter projects more efficiently from a semantic level, improving the transparency and accuracy of search results.

[0117] Step 8. Match the query statement with the project:

[0118] Elasticsearch is used to search the query statement entered by the user on each cluster clustered in step 7. The returned results include all the hit open source projects, the fields in the hit open source projects that match the query statement, and the relevance scores corresponding to the matched fields. Each field includes the field name and its corresponding field value, and the content in the field value that matches the query term is highlighted.

[0119] For each hit open source project, the field with the highest relevance score among all matching fields is selected as the optimal field. Since different arrangement orders of query terms may convey different semantic information, the query statement entered by the user is decomposed into several independent query terms, and the highlighted content in the optimal field is matched with several query terms one by one to obtain the semantic order of the optimal field.

[0120] Furthermore, the step of obtaining the optimal semantic order of the fields in step 8 includes:

[0121] Step 8-1: Label all query terms in the query statement from left to right, and label the highlighted content corresponding to the query term in the optimal field with the same label as the query term.

[0122] Step 8-2: The optimal field includes N sentences, N≥1, and the labels of all query terms in each sentence are formed into an array, and repeated labels in the array are deleted.

[0123] Step 8-3: Connect the N arrays at the end to obtain a one-dimensional array composed of labels. When the elements in the one-dimensional array are repeated periodically from left to right, keep the elements in the first cycle and delete the elements in other cycles to obtain the semantic order of the optimal fields in the query statement in the open source projects hit.

[0124] This will further optimize the relevance assessment of the retrieved items and significantly improve the accuracy of the retrieval results.

[0125] For example, the query is machine learning AI, and the three query words are labeled 0, 1, and 2 respectively. If the optimal field of a hit open source project is readme, which includes 4 sentences, the highlighted field is as follows: Figure 4 . Use curIndex = query_arr.index(curItem) to find the index position of the word in the query word array (query_arr = ["machine", "learning", "ai"]).

[0126] Assume that the query term array is: query_arr = ["machine", "learning", "ai"], then:

[0127] curItem="machine" curIndex="0

[0128] curItem="learning" curIndex=1

[0129] curItem="AI"curIndex="2"

[0130] For a hit open source project, the optimal field has 4 sentences. The labels of all query words in each sentence are used to form an array, that is, four arrays are obtained, namely [0,1], [2,2][0,1,2], [0,1,2]. After removing the repeated labels in each array, we get: [0,1], [2][0,1,2], [0,1,2]. The four arrays are connected to get a one-dimensional array: [0,1,2,0,1,2,0,1,2]. From left to right, periodic repetitions begin from the four elements. The repeated second cycle [0,1,2] and the third cycle [0,1,2] are deleted, and only the first cycle [0,1,2] is retained.

[0131] Step 9: Parse the source code of all hit open source projects:

[0132] For the hit open source projects, Esprima is first used to parse the source code file and generate the corresponding abstract syntax tree AST. By traversing the abstract syntax tree, multiple key identifiers including folder names, file names, function names, and variable names can be extracted, and the frequency of occurrence of these identifiers is counted. In addition, the method further includes calculating the similarity between these key identifiers to achieve the merging of some key identifiers. However, in view of the complexity and time-consuming nature of the similarity calculation process, the merging operation is not performed by default. Based on the parsed key identifiers and their corresponding frequency information, a word cloud is generated for the display of the project details view in step 11. The generated word cloud effectively helps users quickly identify key identifiers and focus on high-frequency content, so that users can intuitively understand the important information of the retrieved project, significantly improving the efficiency of project understanding and analysis.

[0133] Step 10: Recommend open source projects that are related and similar to the hit open source project:

[0134] All hit open source projects are taken as a recommendation list, and other open source projects that are relevant and similar to the hit open source projects are added to the recommendation list to obtain a final recommendation list.

[0135] The recommendation list is used to display the project details view in step 11, providing users with more projects that may be of interest.

[0136] Furthermore, in step 10, the step of obtaining the final recommendation list includes:

[0137] Step 10-1: Use elasticsearch to find other open source projects of the owner or contributor of the hit open source project; if the number of lines of code contributed by each contributor to the hit open source project exceeds the set threshold, the contributor will be included in the contributor search list; if the number of lines contributed by the contributor is lower than this threshold, the first three contributors will be included in the contributor search list by default. Also use elasticsearch to search for other open source projects of the contributors included in the list; define the other open source projects of the owner or contributor found as other related open source projects.

[0138] Step 10-2: Process the project name, description, and README text of each project in other related open source projects [the processing here includes word segmentation, stop word removal, abbreviation replacement, stemming, and part-of-speech restoration] and merge them into a string list. The index of the string list represents the number of each project, and the list elements represent the key information of each project.

[0139] The same method is used to process and merge the project name, description, and README text of each hit open source project into a string list.

[0140] Step 10-3: Calculate the cosine similarity between the hit open source project and other related open source projects, sort them in descending order according to the cosine similarity value, add the top ten related open source projects to the recommendation list, and obtain the final recommendation list.

[0141] Step 11: Visualize and retrieve project data from multiple angles:

[0142] The project data is displayed through visualization technology, including but not limited to the basic information, subject information, dependency information and source code of the project. The visualization process uses multiple perspectives to display the data, and realizes dynamic interaction between users and search results through an interactive operation interface to support users' in-depth exploration and analysis of the search results.

[0143] like Figure 1 As shown, the system of the present invention mainly consists of two parts, including five components, providing users with a complete retrieval paradigm, including four stages of "search-exploration-check-recommendation". The first part ( Figure 1 The A1, A2, A3, B1, B2, C3, C4, and E8 sections of the Figure 1 D) provides a detailed analysis of a single project.

[0144] Basic interface: This component is the core entrance of the system, mainly including the search bar ( Figure 1 - Section A1), filter and sort results ( Figure 1 A2 in the middle), and view switching ( Figure 1 The user can enter keywords or query statements in the search bar to start the search operation. The "Search in result" option is provided to allow users to perform a secondary search in the current search results. Users can filter and sort the results ( Figure 1 In the A2 part of the topic tree, select to display "all" or "part" of the projects in a subnode in the topic tree, and sort the results according to indicators such as the number of stars and forks. The basic information displayed for each project includes the project name, description, tags, and related indicators (such as stars, forks, and attention). When a user clicks on a project, the corresponding project details view will be expanded to display the detailed information of the project. When a user clicks on a project in the A2 subcomponent, the project details view will automatically expand. At this time, the user can collapse and expand the project details view by clicking the A3 button, thereby improving browsing efficiency;

[0145] Topic Overview View: This component displays the topics mined from the retrieved items, including a topic tree ( Figure 1 B1 section) and control module ( Figure 1 The topic tree shows all topics and their hierarchical relationships. Users can click on nodes to select topics of interest. The control module provides options to adjust the layout, style, and parameters of topic modeling. Users can dynamically adjust these settings based on the data scale to optimize the topics and make the view clearer and easier to understand. This view improves the transparency of the results from a semantic perspective and enhances the user's understanding of the search results.

[0146] Topic Cluster View: After selecting a topic, users can view the items under the selected topic. Cluster Chart ( Figure 1 C4 shows the popularity indicators (stars, forks, and followers) of each project. Users can quickly identify more important projects based on color intensity. This view also provides a variety of filtering options, allowing dynamic selection of the presented data. Technical statistics chart ( Figure 1 The C3 section in the cluster shows the frequency of use of all technologies used by the projects in the current cluster. Users can specify specific technologies or exclude irrelevant technologies, and the cluster chart will add or remove projects based on the filter criteria.

[0147] Project Details View: This component is designed to improve the efficiency of users understanding project source code and locating important files. It contains three parts: basic project information ( Figure 1 D1 in the middle), visualization of source code parsing results ( Figure 1 D2) and the Project Recommendation Form ( Figure 1 In the D3 section). Figure 1 In the D2 section, the chart on the left visualizes the frequency of use of four types of identifiers: folder name, file name, function name, and variable name. After clicking on the name, the file structure tree in the middle shows the location of the file. After further selecting the file, the user can view the source code.

[0148] Global Control Panel: In order to provide users with more flexible and diverse interactions, the system supports multiple globally controlled filtering options, including three popularity indicators (stars, forks, and followers), project update time, matching order, and technology-related information such as dependencies, availability, and licenses. The logical relationship between these filtering options is "AND".

[0149] To verify the effectiveness of the invented method, we conducted a user study on 12 experienced JavaScript programming and project retrieval developers. First, we divided them into two groups (G1 and G2) equally and ensured that the experience of the two groups was comparable. G1 used the invented method, and G2 used GitHub's search engine. In order to suppress the influence of subjective bias, each group member was required to complete the same project retrieval task: satellite visualization. Everyone was asked to record the completion time of 8 objective questions, which measured the effectiveness of the invented system in terms of transparency of results, source code understanding, and result extension. Finally, we further interviewed the participants one by one to collect feedback.

[0150] Q1: How many projects have more than 30 stars or followers?

[0151] Q2: How many projects visualize weather data?

[0152] Q3: Find five projects based on the React framework.

[0153] Q4: Find five projects based on Three.js.

[0154] Q5: Find five projects that have examples in their README files.

[0155] Q6: What satellite-related aspects have been achieved by the projects with the highest number of stars?

[0156] Q7: What metrics are calculated in the satellite performance component of the project with the highest number of stars?

[0157] Q8: Find two visualization projects related to satellite tracking.

[0158] Figure 3 The average performance of G1 and G2 on each question is shown. Both groups achieved high accuracy on clear questions, however, G1 performed slightly better on questions that require further exploration (such as Q2, Q5, and Q8). At the same time, the time consumption of G1 was significantly lower than that of G2. The processing time of G1 on all questions was better than that of G2. In particular, when dealing with complex questions, the time consumption of G2 increased significantly, while the time consumption of G1 remained stable. On Q2, Q3, and Q4, the time consumption of G2 was close to 200 seconds, while that of G1 was only about 50 seconds, which fully demonstrated the significant advantages of the new method in high-complexity tasks, indicating that G1 is more efficient and can better cope with complex scenarios. This shows that the system effectively helps users retrieve items. In the interview, they all agreed that the system enhanced the understanding of the results by visualizing themes, making the exploration process more targeted and easier to use. Library dependencies and matching order eliminate unnecessary item checks and improve retrieval efficiency.

Claims

1. A visual retrieval method for open source projects, characterized in that: The steps include: Step 1: Get several open source projects from the open source community: Extract key information for each open source project, including the project name, author information, number of observers, number of stars, number of branches, project description, project size, functional image, license information, contributor information, programming language and its number of code lines, as well as the project's page link and source code link; Step 2: Resolve and organize library dependencies: From each open source project, parse the corresponding package.json file and lib / libs folder to extract the project's dependency information, including dependency name, dependency and its hierarchical structure. By dividing the prefix of the dependency name, extract the hierarchical structure with parent-child node relationship. The extracted dependencies and their hierarchical structure will be written into the predefined {techs} field for subsequent index establishment; Step 3. Extract and categorize the README file: The README file provided by each open source project is used to describe the functions of the open source project. The multi-label classifier READMEClassifier is used to classify the content of each README file into eight categories, namely What, Why, How, When, Who, References, Contribution, and Other. The text classified into the What and Why categories is extracted as the functional text; Step 4. Extract functional images and add usability indicators: The BeautifulSoup library is used to extract the functional images of each open source project. The links of the extracted functional images will be written into the predefined {images} field. The availability of the open source project is defined as whether there is a How category in the README file. If there is a How category, a new field {hasHow:True} is added; otherwise, {hasHow:False} is added. Both the {images} and {hasHow} fields will be used for subsequent indexing. Step 5: Extract topics based on project information BERTopic: Merge the key information of each project to obtain merged information, and train the BERTopic model based on the merged information to extract the topic, topic number, corresponding topic keywords and keyword weights corresponding to the keywords of each open source project. The extracted topic keywords will be recorded in the predefined {topics} field for subsequent index establishment and hierarchical topic modeling. Step 6: Create an index: Input all the information extracted from steps 1 to 5 into Elasticsearch and create an index for each open source project. Step 7: Hierarchical topic modeling: Open source projects with the same topic are classified into the same topic class. Then clustering methods are used to cluster the topic classes. Two topic classes with a distance less than a preset threshold are classified into one cluster. When the topic classes in each cluster no longer change, clustering is completed. Step 8. Match the query statement with the project: Elasticsearch is used to search the query statement entered by the user on each cluster clustered in step 7. The returned results include all the hit open source projects, the fields in the hit open source projects that match the query statement, and the relevance scores corresponding to the matched fields. Each field includes the field name and its corresponding field value, and the content in the field value that matches the query term is highlighted; For each open source project that is hit, the field with the highest relevance score among all matching fields is selected as the optimal field, the query statement entered by the user is decomposed into several independent query terms, and the highlighted content in the optimal field is matched with several query terms one by one to obtain the semantic order of the optimal field; Step 9: Parse the source code of all hit open source projects: For the open source projects that are hit, Esprima is used to parse the source code files and generate the corresponding abstract syntax tree AST; Step 10: Recommend open source projects that are related and similar to the hit open source project: All hit open source projects are taken as a recommendation list, and other open source projects that are relevant and similar to the hit open source projects are added to the recommendation list to obtain a final recommendation list; Step 11: Visualize all the results from step 1 to step 10.

2. The visual retrieval method for open source projects according to claim 1, characterized in that: The step of obtaining the optimal semantic order of fields in step 8 includes: Step 8-1: label all query terms in the query statement from left to right, and label the highlighted content corresponding to the query term in the optimal field with the same label as the query term; Step 8-2, the optimal field includes N sentences, N ≥ 1, the labels of all query terms in each sentence are formed into an array, and the repeated labels in the array are deleted; Step 8-3: Connect the N arrays at the end to obtain a one-dimensional array composed of labels. When the elements in the one-dimensional array are repeated periodically from left to right, keep the elements in the first cycle and delete the elements in other cycles to obtain the semantic order of the optimal fields in the query statement in the open source projects hit.

3. The visual retrieval method for open source projects according to claim 1, characterized in that: In step 10, the step of obtaining the final recommendation list includes: Step 10-1: Use elasticsearch to find other open source projects of the owner or contributor of the hit open source project; if the number of lines of code contributed by each contributor to the hit open source project exceeds the set threshold, the contributor will be included in the contributor search list; if the number of lines of code contributed by the contributor is lower than this threshold, the first three contributors will be included in the contributor search list by default, and elasticsearch will be used to search for other open source projects of the contributors included in the list; define the other open source projects of the owner or contributor found as other related open source projects; Step 10-2: Process and merge the project name, description, and README text of each project in other related open source projects into a string list. The index of the string list represents the number of each project, and the list element represents the key information of each project. The same method is used to process and merge the project name, description, and README text of each hit open source project into a string list; Step 10-3: Calculate the cosine similarity between the hit open source project and other related open source projects, sort them in descending order according to the cosine similarity value, add the top ten related open source projects to the recommendation list, and obtain the final recommendation list.

4. The visual retrieval method for open source projects according to claim 1, characterized in that: In step 11, the visual display includes five components: a basic interface component, a theme overview view component, a theme clustering view component, a project details view component, and a global control panel; Basic interface components: including search bar, result filtering and sorting, and view switching. The search bar is used to enter keywords or query statements to start the search operation; Result filtering and sorting are used to select and display "all" or "part" of the open source projects in a subnode in the topic tree, and sort the open source projects according to the set indicators; view switching is used to automatically expand and collapse the selected open source projects; The topic overview view component is used to display the topic tree of all open source projects in the final recommended project list. The topic overview view component also includes a control module for adjusting parameters; The topic clustering view component is used to display the clustering results in a chart and also to display the usage frequency of the open source projects in the clustering results; The project details view component consists of three parts: basic project information, visualization of source code parsing results, and a project recommendation table; The basic project information is used to display the frequency of use of folder names, file names, function names, and variable names. The visualization of source code analysis results is used to display the source code of all hit open source projects. The project recommendation table is used to display the final recommended columns. Global dashboard component: used to display the popularity index, project update time, and matching order of open source projects in the final recommendation column.

Images