An algorithm for calculating paper similarity based on research fields

Abstract

The present invention relates to the field of paper similarity calculation, and in particular to an algorithm for calculating paper similarity based on research fields, including: S1: downloading papers from a paper library, counting the research fields with the highest frequency in the papers, supplementing the missing parent fields, assigning the same weight to the parent fields as their child fields, and normalizing the weights; S2: representing the research field and related confidence weights of each paper as multiple hierarchical vectors according to the level of the research field, and the multiple hierarchical vectors are the initial representations of the papers; S3: constructing a correlation matrix of the papers, multiplying the correlation matrix by the initial representation of the papers, and obtaining the final representation of the papers in the hierarchy; S4: assigning different weight parameters to each hierarchy, and calculating the similarity between the papers. The similarity between the papers is calculated by combining the coupling degree of the research fields to which the papers belong and the corresponding confidence, which has novelty and good accuracy.

Description

Technical Field

[0001] The present invention relates to the field of paper similarity calculation, and in particular to an algorithm for calculating paper similarity based on research fields. Background Art

[0002] In order to accurately push similar papers to users, calculating paper similarity can accurately find papers similar to the target paper, exclude irrelevant papers, and reduce user search time. Paper similarity calculation is also an important technical basis for knowledge mining. By analyzing the research directions of similar papers, it can predict possible knowledge or topics that will form intersections in the future.

[0003] The most intuitive method is to use natural language processing to process paper text. It extracts features from the title, abstract, and other text information of the paper, and then compares these features to get the results. For text data, N-gram or Word2Vec is generally used to vectorize the text, and then cosine similarity is used to calculate the similarity between vectors to get the similarity between papers. Although the abstract and title can represent the research direction and theme of the paper, due to the word limit, they cannot fully reflect the content of the article, so the extracted features cannot fully represent the paper, which leads to errors in the calculation results.

[0004] Masoud Reyhani Hamedani, Sang-Wook Kim, etc. in Information Science, Volumes 334–335, 2016, Pages 273-292, ISSN 0020-0255 combined the citation relationship of the paper with the text data to calculate the similarity of the paper. However, the calculation of text similarity is still to vectorize the terms in the paper and then measure the similarity of the vectors. The similarity between the papers is finally calculated by quantifying the co-citation relationship between the papers and combining it with the text similarity. A single citation relationship will cause errors in the calculation. Summary of the invention

[0005] The purpose of the present invention is to overcome the problem of large errors in the methods of combining citation relationships and text data in the prior art, and to provide an algorithm for calculating the similarity of papers based on research fields. By combining the coupling degrees of the research fields to which the papers belong and the corresponding confidence levels, a more accurate similarity between the papers can be obtained.

[0006] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0007] An algorithm for calculating paper similarity based on research fields, the algorithm comprising:

[0008] S1: Download papers from the paper database, count the research fields with the highest frequency in the papers, supplement the missing parent fields, assign the same weight to the parent fields as to their child fields, and normalize the weights;

[0009] S2: According to the level of the research field, the research field and the related confidence weight of each of the papers are represented as multiple hierarchical vectors, and the multiple hierarchical vectors are the initial representation of the papers;

[0010] S3: construct a relevance matrix of the paper, and multiply the relevance matrix by the initial representation of the paper to obtain a final representation of the paper at the level;

[0011] S4: assigning different weight parameters to each of the levels, and calculating the similarity between the papers based on the weight parameters.

[0012] The normalization process in S1 includes: increasing the weights of all fields of the paper by 0.1 and performing normalization calculation; the calculation formula is:

[0013] V'=V / |V|

[0014] Among them, V' is the normalized level vector, V is the original level vector, and |V| is the modulus of the level vector.

[0015] The purpose of normalization is to eliminate the dimensional differences between different features and make them comparable.

[0016] Furthermore, the levels in S2 include 6 levels from 0 to 5, and the initial representation of each paper includes 6 level vectors, and the 6 levels and the 6 level vectors correspond one to one.

[0017] Specifically, the correlation matrix in S3 is calculated based on the co-occurrence matrix, and the calculation formula is:

[0018]

[0019]

[0020] Among them, t i，j represents the number of times two research fields co-occur, N i and N j represents the number of times research fields i and j appear respectively, tf ij Indicates the frequency of co-occurrence of research fields i and j, idf ij represents the inverse frequency index of research field i, j, and P represents the number of papers.

[0021] The introduction of the correlation matrix can fully consider the relationship between research fields that affect the similarity of papers, make up for the fact that the similarity defined on the initial representation vector of the paper cannot reflect the subtle differences between fields, and improve the initial representation to contain more information.

[0022] Furthermore, the distribution formula of the weight parameter is:

[0023]

[0024] Among them, L represents the highest level of the research field in the two papers, and i represents the level of the research field.

[0025] As the level of research fields increases, the fields tend to be more specialized. If the high-level fields are similar, then the research directions of the papers are likely to be similar, and the similarity of the papers is higher. Therefore, increasing the weight parameter can distinguish the importance of different levels and more accurately describe the research field of the paper.

[0026] The calculation formula of the similarity is:

[0027]

[0028] Among them, S represents the similarity between papers A and B, Vec_A i , Vec_B i Represents the initial vectors of papers A and B at level i, M i Represents the correlation matrix of level i.

[0029] Each level of the two papers is combined and summarized with the weight parameters to obtain the similarity between the papers. The introduction of levels and weight parameters makes the description of similarity more accurate.

[0030] A computer-readable storage medium stores a computer program, which implements the above algorithm when executed by a processor.

[0031] Compared with the prior art, the present invention has the following beneficial effects:

[0032] The similarity between papers is calculated by combining the coupling degree of the research fields to which the papers belong and the corresponding confidence, which is novel and accurate.

[0033] By representing each paper as a vector of 6 levels, the representation of each paper is made detailed and complete. The similarity of research fields at different levels is used to derive the overall similarity of the papers, thus avoiding excessive influence of the calculation results by fields with a larger generalization scope.

[0034] By introducing a correlation matrix containing the degree of coupling between research fields, the representation vector of the paper is improved so that the vector contains more information and avoids the final similarity being determined only by whether the types of research fields are the same during the calculation process;

[0035] By weighting the results of research fields of different levels and finally integrating the results, the research fields that are more representative of the paper will have greater weight. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the data structure of the method provided by the present invention;

[0037] Figure 2 A schematic diagram of the process of constructing the initial expression vector of the paper according to the method provided by the present invention;

[0038] Figure 3 A schematic diagram of the calculation process of the method provided by the present invention;

[0039] Figure 4 A schematic diagram of calculation results of papers with almost the same research fields as the embodiments provided by the present invention;

[0040] Figure 5 Schematic diagram of calculation results of papers with similar research fields in the embodiments provided by the present invention;

[0041] Figure 6 Schematic diagram of calculation results of papers with very different research fields in the embodiments provided by the present invention. DETAILED DESCRIPTION

[0042] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0043] Example 1

[0044] An algorithm for calculating paper similarity based on research fields, the algorithm comprising:

[0045] S1: Download papers from the paper database, count the research fields with the highest frequency in the papers, supplement the missing parent fields, assign the same weight to the parent fields as to their child fields, and normalize the weights;

[0046] S2: According to the level of the research field, the research field and the related confidence weight of each of the papers are represented as multiple hierarchical vectors, and the multiple hierarchical vectors are the initial representation of the papers;

[0047] S3: construct a relevance matrix of the paper, and multiply the relevance matrix by the initial representation of the paper to obtain a final representation of the paper at the level;

[0048] S4: assigning different weight parameters to each of the levels, and calculating the similarity between the papers based on the weight parameters.

[0049] Download data of 10 million papers from OpenAcademicGraph, first count the 1,000 research fields with the highest frequency in the downloaded papers, and then complete the missing parent research fields of some papers with the same weights of their sub-fields. For example, if a paper has "artificial intelligence" in the first-level field, but does not have "computer science" in the 0-level field (computer science is the parent field of artificial intelligence), then we will add "computer science" to the 0-level field of this paper. Assuming that the weight of "artificial intelligence" is 0.8, the weight of "computer science" is also 0.8.

[0050] In the original data, the fields related to the paper are assigned a weight, which represents the degree of expertise of the paper in the field. However, the weight of the research field of some papers may be 0, which means that the paper is related to the field, but the relevance is not great. In order to distinguish them from those fields that are not related to the paper, the weights of all fields of such papers are increased by 0.1, and finally all weights are normalized. The calculation formula is:

[0051] V'=V / |V|

[0052] Among them, V' is the normalized level vector, V is the original level vector, and |V| is the modulus of the level vector.

[0053] The purpose of normalization is to eliminate the dimensional differences between different features and make them comparable.

[0054] In this example, we select papers that meet the following conditions from these 10 million papers as our experimental dataset: (1) all research fields after completing the parent field are among the top 1000 most frequently occurring fields; (2) in order to understand the research fields of the compared papers in more detail, the papers are required to have at least 4 research fields; (3) the papers were published after 2010.

[0055] Different levels of research fields summarize different ranges, so the correlation values ​​at different levels cannot be mixed together for calculation. Therefore, 1,000 research fields were used, covering levels 0 to 5 based on the research field structure of Microsoft Academic Graph.

[0056] Furthermore, the levels in S2 include 6 levels from 0 to 5, and the initial representation of each paper includes 6 level vectors, and the 6 levels and the 6 level vectors correspond one to one.

[0057] In this example, assuming that the research fields involved in a paper are "medicine", "computer science", "radiology", "machine learning", "diarrhea" and "splicing factor", and the corresponding confidence weights are 0.8, 0.48, 0.3, 0.7, 0.8 and 0.5, the process of constructing the vector is as follows: Figure 2 As shown in the figure. First, the top 1,000 research fields are classified using the research field level data in the Microsoft Academic Graph. Then, the papers can be represented as 6 hierarchical vectors based on the classified research fields. The value of the research field involved in the paper in the vector is the confidence weight of the paper in the field, and the value of the research field not involved in the vector is 0.

[0058] Specifically, the correlation matrix in S3 is calculated based on the co-occurrence matrix, and the calculation formula is:

[0059]

[0060]

[0061] Among them, t i，j represents the number of times two research fields co-occur, N i and N j represents the number of times research fields i and j appear respectively, tf ij Indicates the frequency of co-occurrence of research fields i and j, idf ij represents the inverse frequency index of research field i, j, and P represents the number of papers.

[0062] The correlation matrix is ​​introduced to calculate the co-occurrence times of each pair of research fields of the same level, and then store this information in the form of a co-occurrence matrix. This solves the problem that the distribution of the occurrence times of research fields is uneven and some fields appear too frequently. It can fully consider the relationship between research fields that affect the similarity of papers, make up for the fact that the similarity defined on the initial representation vector of the paper cannot reflect the subtle differences between fields, and improve the initial representation to include more information.

[0063] In this example, the 0-level field of one paper is only "medicine" (which means the confidence weight of "biology" in the paper is 0), and the 0-level field of another paper is only "biology" (which means the confidence weight of "medicine" in the paper is 0), then their cosine similarity is 0, but the relationship between the two disciplines is very close. Since the initial representation is structured, we multiply the correlation matrix by each level of the initial representation to obtain the final representation of the corresponding level of the paper.

[0064] In the final representation of the two papers, the confidence weights corresponding to “biology” in the first paper and “medicine” in the second paper are no longer 0, but are based on the correlation between the two fields, so that their cosine similarity is greater than 0, reflecting the possible similarities between the two papers.

[0065] Furthermore, the distribution formula of the weight parameter is:

[0066]

[0067] Among them, L represents the highest level of the research field in the two papers, and i represents the level of the research field.

[0068] Most papers have very different research fields, so direct comparison may not produce correct results. As the level of research fields increases, the fields tend to be more specialized. If the high-level fields are similar, then the research directions of the papers are likely to be similar, and the papers are more similar. Therefore, increasing the weight parameter can distinguish the importance of different levels and more accurately describe the research fields of the papers.

[0069] Furthermore, the calculation formula of the similarity is:

[0070]

[0071] Among them, S represents the similarity between papers A and B, Vec_A i , Vec_B i Represents the initial vectors of papers A and B at level i, M i Represents the correlation matrix of level i.

[0072] Each level of the two papers is combined and summarized with the weight parameters to obtain the similarity between the papers. The introduction of levels and weight parameters makes the description of similarity more accurate.

[0073] A computer-readable storage medium stores a computer program, which implements the above algorithm when executed by a processor.

[0074] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An algorithm for calculating the similarity of papers based on research fields, characterized in that: The algorithm includes: S1: Count the multiple research fields that appear most frequently in the papers, supplement the missing parent fields, assign the same weights to the parent fields as to their child fields, and normalize the weights; S2: According to the level of the research field, the research field and the related weight of each of the papers are represented as a plurality of level vectors, and the plurality of level vectors are the initial representation of the papers; S3: constructing a correlation matrix of the paper, wherein the correlation matrix is ​​calculated based on the co-occurrence matrix; multiplying the correlation matrix by the initial representation of the paper to obtain the final representation of the paper at the level; S4: assigning different weight parameters to each of the levels, and calculating the similarity between the papers according to the weight parameters; The distribution formula of the weight parameter is: Among them, L represents the highest level of research field in the two papers, and i represents the level of research field; The calculation formula of the similarity is: Among them, Vec_A i , Vec_B i Represents the initial vectors of papers A and B at level i, M i Represents the correlation matrix of level i.

2. The algorithm for calculating paper similarity based on research field according to claim 1 is characterized in that: The normalization process in S1 includes: increasing the weights of all fields of the paper by 0.1 and performing normalization calculation; the calculation formula is: V'=V / |V| Among them, V' is the normalized level vector, V is the original level vector, and |V| is the modulus of the level vector.

3. The algorithm for calculating paper similarity based on research field according to claim 1 is characterized in that: The levels in S2 include 6 levels from 0 to 5, and the initial representation of each paper includes 6 level vectors, and the 6 levels and the 6 level vectors correspond one to one.

4. The algorithm for calculating paper similarity based on research field according to claim 1 is characterized in that: The correlation matrix in S3 is calculated based on the co-occurrence matrix. The calculation formula of the correlation matrix is: Among them, t i，j represents the number of times two research fields co-occur, N i and N j represents the number of times research fields i and j appear respectively, tf ij Indicates the frequency of co-occurrence of research fields i and j, idf ij represents the inverse frequency index of research field i, j, and P represents the number of papers.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the algorithm according to any one of claims 1 to 4 is implemented.

Images