A scientific research topic recommendation method and device based on similarity calculation

Abstract

The present application relates to a kind of research topic recommendation method based on similarity calculation, comprising: obtaining scientific literature data and pre-processing.Utilize TextRank algorithm from abstract and title extract key words, and pass through expert identification technology key word.Use the word2vec algorithm trained to vectorize key word.After obtaining the key word input by user, also vectorize, and calculate the cosine similarity of each vector in key word vector table with it.Select the first few scientific literature and its corresponding scholar and technical key word, and show to user through visualization technology.The present application determines the theme of interest to user by calculating the cosine similarity between key word vectors, to recommend relevant literature, scholar, technology with high relevance to user, help user to quickly find high-quality literature related to the theme of interest to oneself, leading figure in field, possible research method etc..

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and data mining technology, specifically relating to a method and apparatus for recommending research topics based on similarity calculation. Background Technology

[0002] Because academic papers contain a massive amount of data, which is scattered across numerous journals, it is extremely difficult to find a research article that matches one's interests. Therefore, finding articles that match one's interests quickly has become one of the most pressing issues. Summary of the Invention

[0003] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a method and apparatus for recommending scientific research topics based on similarity calculation.

[0004] The objective of this invention can be achieved through the following technical solutions:

[0005] This invention provides a research topic recommendation method based on similarity calculation, comprising the following steps:

[0006] Acquire scientific and technological literature as a dataset, and preprocess the dataset;

[0007] Extract the titles, abstracts, keywords, authors, institutions, and publication dates of scientific and technological literature from the preprocessed dataset;

[0008] Keywords are extracted from abstracts and titles using the TextRank algorithm and combined with keywords extracted from scientific and technological documents to form a keyword list for the scientific and technological documents. Expert experience is then used to identify the corresponding technical keywords for each scientific and technological document from the keyword list.

[0009] The keywords in the keyword list are vectorized using the trained word2vec algorithm to obtain the keyword vector list for each scientific and technological document.

[0010] Get the keywords entered by the user;

[0011] The keywords entered by the user are vectorized using the word2vec algorithm to obtain the keyword vector of the user's input keywords;

[0012] Calculate the cosine similarity between the user's keyword vector and the keyword vector table of each scientific and technological document, and sort the scientific and technological documents according to the cosine similarity.

[0013] The top-ranked scientific and technological documents with the highest cosine similarity are selected as relevant literature recommendations and presented to users using visualization technology.

[0014] Extract scientific and technological documents with a cosine similarity greater than a second preset value to form a relevant scientific and technological document dataset.

[0015] Extract technical keywords from all scientific and technological literature in the relevant scientific and technological literature dataset, count the occurrence of each technical keyword, delete technical keywords whose occurrence is less than the third preset value, sort the technical keywords according to the publication time of their scientific and technological literature, select the technical keywords with the most recent publication time in the fourth preset value as relevant technology recommendations and recommend them to users through visualization technology;

[0016] The number of times each author appears in the relevant scientific literature dataset is counted, and the author with the highest number of appearances (the fifth preset value) is selected as the relevant scholar recommendation and presented to the user through visualization technology.

[0017] Furthermore, the preprocessing of the dataset includes the following steps: cleaning the data, standardizing keywords, author names, institution names, and publication dates.

[0018] Furthermore, the extraction of keywords from the abstract and title using the TextRank algorithm includes the following steps:

[0019] Preprocessing of the abstract and title includes word segmentation, stop word removal, part-of-speech tagging, and converting the text into a format suitable for algorithm processing;

[0020] A graph model is constructed for the preprocessed text, with words or sentences in the processed text as nodes, and the graph model is constructed based on the relationships between words or sentences.

[0021] Calculate the weight of each node;

[0022] The nodes are sorted according to their weights, and the node in the third preset position after sorting is taken as the keyword of the text.

[0023] Furthermore, the calculation of the weight of each node specifically includes: using the iterative idea of ​​the PageRank algorithm to calculate the weight of each node in the graph.

[0024] Furthermore, the relationships between the nodes include co-occurrence relationships and semantic similarity.

[0025] Furthermore, the weight of a node is determined by the weights of other nodes and the strength of the relationships between them.

[0026] Furthermore, the technical keywords are those keywords in the keyword list of this scientific and technological document that best reflect the main technologies of this scientific and technological document.

[0027] Furthermore, the calculation of the cosine similarity between the user's keyword vector and the keyword vector table of each scientific and technological document specifically includes: calculating the cosine similarity between the user's keyword vector and each keyword vector in the keyword vector table of the scientific and technological document, and taking the average value to obtain the cosine similarity between the user's keyword vector and the keyword vector table of the scientific and technological document.

[0028] Furthermore, the cosine similarity between the user keyword vector and each keyword vector in the scientific literature keyword vector table is calculated using the following formula:

[0029]

[0030] in, For the user's keyword vector, This refers to the keyword vectors in the keyword vector table. Let be the cosine similarity.

[0031] The present invention also provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, it implements the scientific research topic recommendation method based on similarity calculation as described in any one of the above.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] (1) This invention vectorizes the keywords of scientific and technological literature using the word2vec algorithm. By calculating the cosine similarity between vectors, it determines the topics of interest to users and recommends highly relevant literature, scholars, technologies, etc., helping users quickly find high-quality literature, leading tasks in the field, and possible research methods related to their topics of interest. Through the topic recommendation function, users can quickly understand the research status of the topic, assess the feasibility of the topic research, determine the direction of the topic research, and improve the quality of topic research.

[0034] (2) This invention uses the TextRank algorithm to extract keywords, which can capture the correlation between words and automatically extract keywords without manual annotation. Attached Figure Description

[0035] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations to adapt to distributed computing architectures such as Particular or other similar or different distributed architectures (such as end-to-end architectures). Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention. It should be noted that similar reference numerals and letters in the following drawings denote similar items; therefore, once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings. In the description of this application, it should also be noted that, unless otherwise expressly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0038] Example 1:

[0039] like Figure 1 As shown in the embodiment, a research topic recommendation method based on similarity calculation is provided, including the following steps:

[0040] S1. Obtain scientific and technological literature as a dataset and preprocess the dataset;

[0041] S2. Extract the titles, abstracts, keywords, authors, institutions, and publication dates of scientific and technological documents from the preprocessed dataset;

[0042] S3. Use the TextRank algorithm to extract keywords from the abstract and title, and combine them with the keywords extracted from the scientific and technological documents to form the keyword list of the scientific and technological documents. Use expert experience to identify the corresponding technical keywords for each scientific and technological document from the keyword list. The technical keywords are the keywords in the keyword list of the scientific and technological document that best reflect the main technology of the scientific and technological document.

[0043] S4. Vectorize the keywords in the keyword list using the trained word2vec algorithm to obtain the keyword vector table for each scientific and technological document.

[0044] S5. Obtain the keywords entered by the user;

[0045] S6. Vectorize the keywords entered by the user using the word2vec algorithm to obtain the keyword vector of the keywords entered by the user;

[0046] S7. Calculate the cosine similarity between the user's keyword vector and the keyword vector table of each scientific and technological document, and sort the scientific and technological documents according to the cosine similarity; specifically, the calculation of the cosine similarity between the user's keyword vector and the keyword vector table of each scientific and technological document includes: calculating the cosine similarity between the user's keyword vector and each keyword vector in the keyword vector table of the scientific and technological document, and taking the average value to obtain the cosine similarity between the user's keyword vector and the keyword vector table of the scientific and technological document.

[0047] S8. Select the top-preset-value scientific and technological literature with the highest cosine similarity as relevant literature recommendations and recommend them to users through visualization technology;

[0048] S9. Extract scientific and technological documents with a cosine similarity greater than the second preset value to form a relevant scientific and technological document dataset.

[0049] S10. Extract the technical keywords of all scientific and technological documents in the relevant scientific and technological literature dataset, count the occurrence of each technical keyword, delete the technical keywords whose occurrence is less than the third preset value, sort the technical keywords according to the publication time of their scientific and technological documents, select the technical keywords with the most recent publication time in the fourth preset value as relevant technical recommendations and recommend them to users through visualization technology.

[0050] S11. Count the number of times each author appears in the relevant scientific and technological literature dataset, and select the author with the highest number of appearances (the fifth preset value) as the relevant scholar recommendation and recommend it to the user through visualization technology.

[0051] Specifically, by preprocessing scientific and technological literature data, some irrelevant content in the text can be eliminated and the amount of text can be reduced, which facilitates the development of subsequent technologies, improves the effectiveness of the technologies, and ensures the feasibility of subsequent technology implementation, thereby ensuring the stability of later technologies, improving the overall level of technological development, and thus helping to promote the continuous improvement of the level of scientific and technological development.

[0052] In S1, duplicate articles in the dataset are removed. Each article is named according to a standardized format, and keywords are standardized. The data is then categorized and organized. Next, the titles and abstracts are segmented using the jieba Chinese word segmentation engine and the nltk natural language processing package, respectively, and stop word removal is performed. The data is then converted into a format suitable for algorithm processing. Word segmentation is the process of treating a Chinese character or multiple characters as a word, while the set of symbols used to convert Chinese characters into syllables is called a pinyin scheme. StopWords is a list of strings containing some commonly used but meaningless phrases; removing it facilitates better analysis.

[0053] Next, the raw data is imported into a deep learning library, PyTorch, and the TextRank algorithm is used for keyword selection. The specific process is as follows: First, a node matrix is ​​created for the articles to be retrieved. Each element in this node matrix is ​​a document, and each document has its own tag attributes (such as ID, category, etc.). Second, a probabilistic graphical model is constructed using a single-layer GloVe neural network model, and weights and biases are assigned to it. Finally, a suitable parameter is obtained by training the model so that the model has good generalization and expressive ability.

[0054] The feature engineering for documents mainly includes the following parts: (1) Word frequency statistics: arrange the words in each document according to their frequency; (2) Deletion of stop words: delete adverbs in the document that are not nouns and frequently appear in news media, such as "today", "just now", "soon", etc., because such words are not very meaningful for subsequent information retrieval; (3) Feature combination: select the TF-IDF model, which is a comprehensive statistical method; (4) Context information matching: if the IDF value of a keyword is positive and the IDF values ​​of its neighboring keywords are negative, then it can be considered a low-frequency keyword, and vice versa. The final table form obtained by the feature engineering methods mentioned above is the document keyword matrix. In this step, it is necessary to determine the required field names and their order, so as to facilitate the subsequent encoding operation.

[0055] The formula for calculating cosine similarity as described in S7 is:

[0056]

[0057] in, For the user's keyword vector, These are vectors from the keyword vector table. Let be the cosine similarity.

[0058] In S8, the similarity between an existing keyword and a keyword of the technology under study is determined by the distance between them. The smaller the distance between a new technology keyword and another reported new technology, the closer they are; conversely, the larger the distance between two new technologies, the further apart they are. Since this embodiment uses the Euclidean distance between vectors to measure the distance between two features, this process is also called the L2 norm. Generally, as the L2 norm increases, it becomes more difficult to distinguish which of all adjacent vectors are truly related, meaning more noise will enter the results. The core of this invention lies in fully utilizing the characteristics of existing scientific and technological knowledge resources, especially information such as the impact factor of the journals in which papers are published and the academic affiliations of scholars. With the help of this existing knowledge, the paper screening and classification work is automatically completed, thereby significantly reducing the time cost of manually searching and reading a large number of references, shortening the paper writing cycle, and greatly improving quality and efficiency.

[0059] Example 2:

[0060] The parts not mentioned in this embodiment are the same as in Embodiment 1.

[0061] This embodiment provides a terminal, which includes a processor. The processor is used to execute a computer program stored in a memory to implement the research topic recommendation method based on similarity calculation.

[0062] This embodiment also provides a computer-readable storage medium storing a computer program, characterized in that the computer program, when executed by a processor, implements the research topic recommendation method based on similarity calculation.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A research topic recommendation method based on similarity calculation, characterized in that, Includes the following steps: Acquire scientific and technological literature as a dataset, and preprocess the dataset; Extract the titles, abstracts, keywords, authors, institutions, and publication dates of scientific and technological literature from the preprocessed dataset; Keywords are extracted from abstracts and titles using the TextRank algorithm and combined with keywords extracted from scientific and technological documents to form a keyword list for the scientific and technological documents. Expert experience is then used to identify the corresponding technical keywords for each scientific and technological document from the keyword list. The keywords in the keyword list are vectorized using the trained word2vec algorithm to obtain the keyword vector list for each scientific and technological document. Get the keywords entered by the user; The keywords entered by the user are vectorized using the word2vec algorithm to obtain the keyword vector of the user's input keywords; Calculate the cosine similarity between the user's keyword vector and the keyword vector table of each scientific and technological document, and sort the scientific and technological documents according to the cosine similarity. The top-ranked scientific and technological documents with the highest cosine similarity are selected as relevant literature recommendations and presented to users using visualization technology. Extract scientific and technological documents with a cosine similarity greater than a second preset value to form a relevant scientific and technological document dataset. Extract technical keywords from all scientific and technological literature in the relevant scientific and technological literature dataset, count the occurrence of each technical keyword, delete technical keywords whose occurrence is less than the third preset value, sort the technical keywords according to the publication time of their scientific and technological literature, select the technical keywords with the most recent publication time in the fourth preset value as relevant technology recommendations and recommend them to users through visualization technology; The number of times each author appears in the relevant scientific literature dataset is counted, and the author with the highest number of appearances (the fifth preset value) is selected as the relevant scholar recommendation and presented to the user through visualization technology.

2. The research topic recommendation method based on similarity calculation according to claim 1, characterized in that, The preprocessing of the dataset includes the following steps: cleaning the data, standardizing keywords, author names, institution names, and publication dates.

3. The research topic recommendation method based on similarity calculation according to claim 1, characterized in that, The method of extracting keywords from the abstract and title using the TextRank algorithm includes the following steps: Preprocessing of the abstract and title includes word segmentation, stop word removal, part-of-speech tagging, and converting the text into a format suitable for algorithm processing; A graph model is constructed for the preprocessed text, with words or sentences in the processed text as nodes, and the graph model is constructed based on the relationships between words or sentences. Calculate the weight of each node; The nodes are sorted according to their weights, and the node in the third preset position after sorting is taken as the keyword of the text.

4. The research topic recommendation method based on similarity calculation according to claim 3, characterized in that, The calculation of the weight of each node specifically includes: using the iterative idea of ​​the PageRank algorithm to calculate the weight of each node in the graph.

5. The research topic recommendation method based on similarity calculation according to claim 3, characterized in that, The relationships between the nodes include co-occurrence relationships and semantic similarity.

6. The research topic recommendation method based on similarity calculation according to claim 5, characterized in that, The weight of a node is determined by the weights of other nodes and the strength of the relationships between them.

7. The research topic recommendation method based on similarity calculation according to claim 1, characterized in that, The technical keywords are those keywords in the keyword list of this scientific and technological document that best reflect the main technologies of this scientific and technological document.

8. The research topic recommendation method based on similarity calculation according to claim 1, characterized in that, The calculation of the cosine similarity between the user's keyword vector and the keyword vector table of each scientific and technological document specifically includes: calculating the cosine similarity between the user's keyword vector and each keyword vector in the keyword vector table of the scientific and technological document, and taking the average value to obtain the cosine similarity between the user's keyword vector and the keyword vector table of the scientific and technological document.

9. A research topic recommendation method based on similarity calculation according to claim 8, characterized in that, The cosine similarity between the user keyword vector and each keyword vector in the scientific literature keyword vector table is calculated using the following formula: in, For the user's keyword vector, This refers to the keyword vectors in the keyword vector table. Let be the cosine similarity.

10. A computer-readable storage medium storing a computer program thereon, characterized in that, When the computer program is executed by the processor, it implements the research topic recommendation method based on similarity calculation as described in any one of claims 1 to 9.

Images