Scientific research topic recommendation method and device based on similarity calculation

Abstract

The invention relates to a similarity calculation-based scientific research topic selection recommendation method. The method comprises the steps of obtaining scientific and technical literature data and performing preprocessing; and extracting keywords from the abstract and the title by using a TextRank algorithm, and identifying technical keywords through experts. And carrying out vectorization on the keywords by using the trained word2vec algorithm. After keywords input by a user are obtained, vectorization is also carried out, and the cosine similarity between the keywords and each vector in the keyword vector table is calculated; and selecting the previous scientific and technical literatures and the corresponding scholars and technical keywords, and displaying the scientific and technical literatures and the corresponding scholars and technical keywords to the user through a visualization technology. According to the method, by calculating the cosine similarity between the keyword vectors, the theme in which the user is interested is determined, literatures, scholars, technologies and the like with high relevancy are recommended for the user, and the user is helped to quickly find high-quality literatures related to the theme in which the user is interested, leading figures in the field, possible research methods and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing and data mining, and specifically relates to a method and device for recommending scientific research topics based on similarity calculation. Background Art

[0002] Since the amount of data in academic papers is huge and the data is scattered in many journals, it is very difficult to find a scientific research academic article that matches your interests; how to quickly retrieve articles that match your interests has become one of the most pressing issues at present. Summary of the invention

[0003] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and device for recommending scientific research topics based on similarity calculation.

[0004] The purpose of the present invention can be achieved by the following technical solutions:

[0005] On the one hand, the present invention provides a method for recommending scientific research topics based on similarity calculation, comprising the following steps:

[0006] Obtain scientific and technological literature as a data set and preprocess the data set;

[0007] Extract the title, abstract, keywords, author, institution and publication time of scientific literature from the preprocessed data set;

[0008] The TextRank algorithm is used to extract keywords from abstracts and titles, and together with the keywords extracted from scientific literature, the keyword table of scientific literature is formed. Expert experience is used to identify the corresponding technical keywords of each scientific literature from the keyword table.

[0009] The keywords in the keyword table are vectorized through the trained word2vec algorithm to obtain the keyword vector table of each scientific document;

[0010] Get the keywords entered by the user;

[0011] Vectorize the keywords input by the user through the word2vec algorithm to obtain the keyword vector of the keyword input by the user;

[0012] Calculate the cosine similarity between the user's keyword vector and the keyword vector table of each scientific document, and sort the scientific documents according to the cosine similarity;

[0013] Select the scientific and technological documents with the highest cosine similarity before the first preset value as the relevant document recommendation and recommend them to the user through visualization technology;

[0014] Extract scientific and technological documents whose cosine similarity is greater than a second preset value to form a relevant scientific and technological document data set;

[0015] Extract technical keywords from all scientific and technological documents in the relevant scientific and technological document data set, count the number of occurrences of each technical keyword, delete the technical keywords whose number of occurrences is less than the third preset value, sort the technical keywords according to the publication time of their scientific and technological documents, and select the technical keywords with the most recent publication time before the fourth preset value as relevant technology recommendations to recommend to users through visualization technology;

[0016] The number of occurrences of each author in the relevant scientific literature data set is counted, and the authors with the largest number of occurrences in the top five preset positions are selected as relevant scholars and recommended to users through visualization technology.

[0017] Furthermore, the preprocessing of the data set includes the following steps: cleaning data, standardizing keywords, author names, institution names and publication time.

[0018] Furthermore, the method of extracting keywords from the abstract and title using the TextRank algorithm includes the following steps:

[0019] Preprocess the abstract and title, including word segmentation, stop word removal, and part-of-speech tagging, to convert the text into a format suitable for algorithm processing;

[0020] Construct a graph model for the preprocessed text, use the words or sentences in the processed text as nodes, and construct a graph model based on the relationship between the words or sentences;

[0021] Calculate the weight of each node;

[0022] The nodes are sorted according to their weights, and the nodes with the third preset value position before the sorting are used as keywords of the text.

[0023] Furthermore, the calculation of the weight of each node specifically includes: using the iterative idea of ​​the PageRank algorithm to calculate the weight of each node in the graph.

[0024] Furthermore, the relationship between the nodes includes co-occurrence relationship and semantic similarity.

[0025] Furthermore, the weight of the node is determined by the weights of other nodes and the strength of the relationship between them.

[0026] Furthermore, the technical keywords are the keywords in the keyword table of the scientific document that best reflect the main technologies of the scientific document.

[0027] Furthermore, the method of calculating the cosine similarity between the user's keyword vector and the keyword vector table of each scientific document specifically includes: calculating the cosine similarity between the user's keyword vector and each keyword vector in the scientific document keyword vector table, and taking the average value to obtain the cosine similarity between the user's keyword vector and the keyword vector table of the scientific document.

[0028] Furthermore, the cosine similarity between the user keyword vector and each keyword vector in the scientific literature keyword vector table is calculated using the following formula:

[0029]

[0030] in, is the user's keyword vector, is the keyword vector in the keyword vector table, is the cosine similarity.

[0031] The present invention also provides a computer-readable storage medium, on which a computer program is stored, characterized in that when the computer program is executed by a processor, the method for recommending scientific research topics based on similarity calculation as described in any one of the above items is implemented.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] (1) The present invention vectorizes the keywords of scientific and technological literature through the word2vec algorithm, and calculates the cosine similarity between vectors to determine the topics that users are interested in, and recommends highly relevant documents, scholars, technologies, etc. to users, helping users quickly find high-quality documents related to the topics they are interested in, leading tasks in the field, possible research methods, etc. Through the topic recommendation function, users can quickly understand the research status of the topic, evaluate the feasibility of the topic research, determine the direction of the topic research, and improve the quality of the topic research.

[0034] (2) The present invention uses the TextRank algorithm to extract keywords, which can capture the correlation between words and automatically extract keywords without manual annotation. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. The components of the embodiment of the present invention described and shown in the drawings here can be arranged and designed in various different configurations to adapt to distributed computing architectures such as particular or other distributed architectures of the same or different modes (such as end-to-end architecture). Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the invention claimed for protection, but only represents the selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings. In the description of this application, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0038] Embodiment 1:

[0039] like Figure 1 As shown, the embodiment provides a method for recommending scientific research topics based on similarity calculation, comprising the following steps:

[0040] S1. Obtain scientific and technological literature as a data set and preprocess the data set;

[0041] S2, extract the title, abstract, keywords, author, institution and publication time of scientific literature from the preprocessed data set;

[0042] S3. Use the TextRank algorithm to extract keywords from the abstract and title, and together with the keywords extracted from the scientific literature, form the keyword table of the scientific literature. Use expert experience to identify the corresponding technical keywords of each scientific literature from the keyword table; technical keywords are the keywords in the keyword table of the scientific literature that best reflect the main technology of the scientific literature.

[0043] S4, vectorize the keywords in the keyword table through the trained word2vec algorithm to obtain the keyword vector table of each scientific document;

[0044] S5, obtaining keywords input by the user;

[0045] S6, vectorizing the keyword input by the user through the word2vec algorithm to obtain a keyword vector of the keyword input by the user;

[0046] S7. Calculate the cosine similarity between the user's keyword vector and the keyword vector table of each scientific document, and sort the scientific documents according to the cosine similarity; calculate the cosine similarity between the user's keyword vector and the keyword vector table of each scientific document, specifically including: calculate the cosine similarity between the user's keyword vector and each keyword vector in the keyword vector table of the scientific document, and take the average value to obtain the cosine similarity between the user's keyword vector and the keyword vector table of the scientific document.

[0047] S8, selecting the scientific and technological documents with the highest cosine similarity before the first preset value as relevant documents and recommending them to the user through visualization technology;

[0048] S9, extracting scientific and technological documents whose cosine similarity is greater than a second preset value to form a relevant scientific and technological document data set;

[0049] S10, extracting technical keywords from all scientific and technological documents in the relevant scientific and technological document data set, counting the number of occurrences of each technical keyword, deleting the technical keywords whose number of occurrences is less than the third preset value, and sorting the technical keywords according to the publication time of their scientific and technological documents, selecting the technical keywords with the most recent publication time before the fourth preset value as relevant technology recommendations and recommending them to the user through visualization technology;

[0050] S11. Count the number of occurrences of each author in the relevant scientific literature data set, and select the authors with the largest number of occurrences in the top five preset positions as relevant scholars to recommend to users through visualization technology.

[0051] Specifically, through the preprocessing operation of scientific and technological literature data, some irrelevant content in the text can be eliminated and the number of words can be reduced, which is convenient for the development of subsequent technologies, improves the effectiveness of technologies, and ensures the feasibility of subsequent technology implementation, thereby ensuring the stability of subsequent technologies and improving the development level of the entire technology, which in turn helps to promote the continuous improvement of the level of scientific and technological development.

[0052] In S1, duplicate articles in the data set are removed, each article in the data set is named in a unified format, keywords are standardized, and then the data are classified and sorted; then the title and abstract are segmented and stop words are removed using the jieba Chinese word segmentation engine and the nltk natural language analysis package, and the data are converted into a format suitable for algorithm processing. Among them, word segmentation is the process of treating a Chinese character or multiple characters as a word, and the symbol set that converts Chinese into syllables is called the pinyin scheme. StopWords is a string list containing some commonly used meaningless phrases, so removing it is conducive to better analysis.

[0053] After that, the original data is imported into a deep learning library pytorch, and the TextRank algorithm is used to select keywords. The specific process is as follows: First, a node matrix is ​​established for the article to be retrieved. Each element in this node matrix is ​​a document, and each document has its label attributes (such as ID, category, etc.); secondly, a single-layer GloVe neural network model is used to build a probabilistic graph model, and weights and biases are assigned to it; finally, a suitable parameter is obtained by training the model so that the model has good generalization and expression capabilities.

[0054] The feature engineering for documents mainly includes the following parts: (1) word frequency statistics: arrange the words appearing in each document according to their frequency; (2) delete stop words: delete adverbs in the document that are not nouns and often appear in news media, such as "today", "just now", "about to", etc., because such words are not very meaningful for subsequent information retrieval; (3) feature combination: select the TF-IDF model, which is a comprehensive statistical method; (4) context information matching, if the IDF value of a keyword is positive, and the IDF values ​​of its neighboring keywords are all negative, then it can be considered that this is a low-frequency keyword, otherwise it is not. The table form finally obtained by the above-mentioned feature engineering processing methods is the document keyword matrix. In this step, it is necessary to determine the required field names and their order, so as to facilitate the subsequent encoding operation.

[0055] The calculation formula of cosine similarity described in S7 is:

[0056]

[0057] in, is the user's keyword vector, is the vector in the keyword vector table, is the cosine similarity.

[0058] In S8, the distance between an existing keyword and the technical keyword to be studied is used to judge the similarity between the two. The smaller the distance between a new technology keyword and another new technology that has been reported, the closer the two are; on the contrary, if the distance of a new technology is larger, it means that the two technologies are farther apart. Since the present embodiment uses the Euclidean distance between vectors to measure the distance between two features, this process is also called the L2 norm. In general, as the L2 norm increases, it becomes more difficult to distinguish which two adjacent vectors are truly related, which means that more noise will enter the results. The core of the present invention is to make full use of the characteristics of existing scientific and technological knowledge resources, especially the impact factors of journals where papers are published, the work units of scholars and other information, and with the help of these existing knowledge, automatically complete the work of paper screening and classification, thereby greatly reducing the time cost of manually searching and reading a large amount of reference materials, shortening the paper writing cycle while greatly improving the quality and efficiency.

[0059] Embodiment 2:

[0060] The parts not mentioned in this embodiment are the same as those in Embodiment 1.

[0061] This embodiment provides a terminal, which includes a processor, and the processor is used to implement the scientific research topic recommendation method based on similarity calculation when executing a computer program stored in a memory.

[0062] This embodiment also provides a computer-readable storage medium, on which a computer program is stored, and it is characterized in that when the computer program is executed by a processor, the method for recommending scientific research topics based on similarity calculation is implemented.

[0063] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0064] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for recommending scientific research topics based on similarity calculation, characterized in that: The following steps are involved: Obtain scientific and technological literature as a data set and preprocess the data set; Extract the title, abstract, keywords, author, institution and publication time of scientific literature from the preprocessed data set; The TextRank algorithm is used to extract keywords from abstracts and titles, and together with the keywords extracted from scientific literature, the keyword table of scientific literature is formed. Expert experience is used to identify the corresponding technical keywords of each scientific literature from the keyword table. The keywords in the keyword table are vectorized through the trained word2vec algorithm to obtain the keyword vector table of each scientific document; Get the keywords entered by the user; Vectorize the keywords input by the user through the word2vec algorithm to obtain the keyword vector of the keyword input by the user; Calculate the cosine similarity between the user's keyword vector and the keyword vector table of each scientific document, and sort the scientific documents according to the cosine similarity; Select the scientific and technological documents with the highest cosine similarity before the first preset value as the relevant document recommendation and recommend them to the user through visualization technology; Extract scientific and technological documents whose cosine similarity is greater than a second preset value to form a relevant scientific and technological document data set; Extract technical keywords from all scientific and technological documents in the relevant scientific and technological document data set, count the number of occurrences of each technical keyword, delete the technical keywords whose number of occurrences is less than the third preset value, sort the technical keywords according to the publication time of their scientific and technological documents, and select the technical keywords with the most recent publication time before the fourth preset value as relevant technology recommendations to recommend to users through visualization technology; The number of occurrences of each author in the relevant scientific literature data set is counted, and the authors with the largest number of occurrences in the top five preset positions are selected as relevant scholars and recommended to users through visualization technology.

2. According to claim 1, a method for recommending scientific research topics based on similarity calculation is characterized in that: The preprocessing of the data set includes the following steps: cleaning data, standardizing keywords, author names, institution names and publication time.

3. A method for recommending scientific research topics based on similarity calculation according to claim 1, characterized in that: The method of extracting keywords from the abstract and title using the TextRank algorithm includes the following steps: Preprocess the abstract and title, including word segmentation, stop word removal, and part-of-speech tagging, to convert the text into a format suitable for algorithm processing; Construct a graph model for the preprocessed text, use the words or sentences in the processed text as nodes, and construct a graph model based on the relationship between the words or sentences; Calculate the weight of each node; The nodes are sorted according to their weights, and the nodes with the third preset value position before the sorting are used as keywords of the text.

4. A method for recommending scientific research topics based on similarity calculation according to claim 3, characterized in that: The calculating of the weight of each node specifically includes: using the iterative idea of ​​PageRank algorithm to calculate the weight of each node in the graph.

5. According to the method for recommending scientific research topics based on similarity calculation in claim 3, it is characterized in that: The relationship between the nodes includes co-occurrence relationship and semantic similarity.

6. A method for recommending scientific research topics based on similarity calculation according to claim 5, characterized in that: The weight of the node is determined by the weights of other nodes and the strength of the relationship between them.

7. A method for recommending scientific research topics based on similarity calculation according to claim 1, characterized in that: The technical keywords are the keywords in the keyword table of the scientific document that best reflect the main technologies of the scientific document.

8. The method for recommending scientific research topics based on similarity calculation according to claim 1, characterized in that: The method of calculating the cosine similarity between the user's keyword vector and the keyword vector table of each scientific document specifically includes: calculating the cosine similarity between the user's keyword vector and each keyword vector in the scientific document keyword vector table, and taking the average value to obtain the cosine similarity between the user's keyword vector and the keyword vector table of the scientific document.

9. A method for recommending scientific research topics based on similarity calculation according to claim 8, characterized in that: The calculation formula for calculating the cosine similarity between the user keyword vector and each keyword vector in the scientific literature keyword vector table is: in, is the user's keyword vector, is the keyword vector in the keyword vector table, is the cosine similarity.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for recommending scientific research topics based on similarity calculation as described in any one of claims 1 to 9.

Images