Subject field division method based on hierarchical clustering of paper keywords and semantics

Abstract

The invention discloses a subject field division method based on hierarchical clustering of paper keywords and semanteme, which comprises the following steps of: retrieving all keywords of a set subject publication published paper based on a big data technology, and preprocessing all the keywords to obtain preprocessed keywords; analyzing semantics based on the preprocessed keywords to obtain semantic information, and classifying the keywords based on the semantic information to obtain a plurality of categories; setting the number of categories, and performing keyword clustering analysis on the data in the plurality of categories based on the number of categories to obtain a clustering result; performing category division on the subject field based on the clustering result to obtain a subject field division result, and transmitting the subject field division result to the terminal in real time; according to the method and the device, the requirement of division of different subdivision fields of the subject can be met, so that subject evaluation and subject talent evaluation are finer and more accurate, and the distinctive subject fields and subject field talents are found.

Description

Technical Field

[0001] The present invention relates to the technical field of subject division, and more specifically, to a subject field division method based on hierarchical clustering of paper keywords and semantics. Background Art

[0002] With the development of big data technology and artificial intelligence technology, third-party evaluation is increasingly valued by society, government and enterprises. However, in the current subject division, most subjects have a wide coverage. For example, the Springer system abroad divides journals and books into 13 subjects: architecture and design, behavioral sciences, biomedicine and life sciences, business and economics, chemistry and materials science, computer science, earth and environmental sciences, engineering, humanities, social sciences and law, mathematics and statistics, medicine, physics and astronomy, computer vocational technology and professional computer applications. In China, subjects are divided into 14 disciplines, including philosophy, economics, law, education, literature, history, science, engineering, agriculture, medicine, military science, management, art, and interdisciplinary subjects, with a total of 113 first-level disciplines; these "disciplines" are basically set up according to the classification and degree authorization of postgraduates, and are universal and broad, and appear to be very broad in specific subject evaluation and talent evaluation. For example, in the science category, there are first-level disciplines such as mathematics, physics, chemistry, astronomy, geography, atmospheric science, marine science, geophysics, and geology. In fact, the scope of disciplines such as mathematics and physics is very broad. For example, a scholar is very outstanding in the research of "Hall quantum effect", but he may not be very outstanding in the whole physics; similarly, a university has outstanding research results in "Hall quantum effect", but it may not be very good in the evaluation of physics. This situation has brought great limitations to the evaluation of disciplines and talents, especially in the introduction of talents, which often requires special talents in a certain aspect of the discipline. In this way, it is particularly necessary to divide the discipline into sub-fields.

[0003] When determining the academic level and technical ability of the evaluated talents in the discipline, if we compare them from the entire discipline, there are so many talents that it is difficult to reflect their characteristics and find talents who truly have special research capabilities in a certain research direction of the discipline. Summary of the invention

[0004] In order to solve at least one of the above technical problems, the present invention proposes a subject field division method based on hierarchical clustering of paper keywords and semantics, so as to realize the application of big data technology to retrieve keywords in all papers of a given subject, hierarchically cluster the keywords, and use the clustered categories as the sub-fields of the subject, thereby forming a paper collection and keyword collection in the sub-field.

[0005] The first aspect of the present invention provides a method for dividing subject areas based on hierarchical clustering of paper keywords and semantics, comprising the following steps:

[0006] Based on big data technology, all keywords of publicly published papers in a given discipline are retrieved, and all keywords are preprocessed to obtain preprocessed keywords;

[0007] Analyze the semantics of the preprocessed keywords to obtain semantic information, and classify the keywords based on the semantic information to obtain multiple categories;

[0008] Set the number of categories, perform keyword clustering analysis on the data in multiple categories based on the number of categories, and obtain clustering results;

[0009] Based on the clustering results, the subject fields are classified to obtain the subject field division results, and the subject field division results are transmitted to the terminal in real time.

[0010] In a preferred embodiment of the present invention, all keywords of publicly published papers in a set discipline are retrieved based on big data technology, and all keywords are preprocessed to obtain preprocessed keywords, which specifically include:

[0011] Determine the search subject, use big data technology to retrieve all publicly published papers in the subject, and obtain the keywords of each paper;

[0012] Duplicate keywords of each paper are identified, and the identified identical keywords are merged to obtain the preprocessed keywords.

[0013] In a preferred embodiment of the present invention, semantics are analyzed based on the preprocessed keywords to obtain semantic information, and keywords are classified based on the semantic information to obtain multiple categories, specifically including:

[0014] Apply big data technology to retrieve the number of times each keyword appears in each paper, take each keyword as a feature vector, and the number of times it appears in each paper as the dimension value to construct the keyword feature vector;

[0015] Perform word frequency-inverse document word frequency processing on the frequency of keywords in each paper, and establish the paper keyword word frequency-inverse document word frequency feature vector;

[0016] Calculate the total word frequency of each keyword in all papers - inverse document word frequency, sort from large to small, and take the keywords with the set percentage sequence of keywords as the core words of the paper;

[0017] Analyze the semantics of the core words of the paper to obtain semantic information. Based on the semantic information and the core words of the paper, use them as the names of subdivided fields, and use the number of core words as the number of clusters to obtain multiple categories.

[0018] In a preferred embodiment of the present invention, a feature vector of the frequency of occurrence of keywords in a paper is established, which specifically includes:

[0019] Obtain data on M papers and N keywords, and retrieve the number of times each keyword appears in each paper based on big data technology;

[0020] Taking each keyword as a feature vector and the number of times it appears in each paper as the dimension value, the constructed keyword feature vector is:

[0021] Keyword K1=[k 11 , k 12 ,......,k 1M ]

[0022] Keyword K2 = [k 21 , k 22 ,......,k 2M ] ......

[0024] Keyword K N =[k N1 , k N2 ,......,k NM ].

[0025] In a preferred embodiment of the present invention, a paper keyword frequency-inverse document frequency feature vector is established, specifically including: after constructing the paper keyword occurrence frequency feature vector, eliminating the influence of common words on the frequency, and performing word frequency-inverse document frequency processing on the frequency of keyword occurrence in each paper, and the calculation formula is:

[0026]

[0027] In the formula, k ij is the number of times keyword i appears in article j, is the total number of times keyword i appears in all papers, log is the logarithm, D is the total number of papers in the subject, and D i The number of papers in which keyword i appears in the subject papers is used to establish the feature vector of keyword frequency-inverse document frequency:

[0028] Keyword Ks1=[S 11 , S 12 ,,.......,S 1M ]

[0029] Keyword Ks2=[S 21 , S 22 ,,.......,S 2M ] ......

[0031] Keyword Ks N =[S N1 , S N2 ,,.......,S NM ].

[0032] In a preferred embodiment of the present invention, the method for determining the core words of the paper and the number of clusters is as follows:

[0033] After the paper keyword frequency-inverse document frequency feature vector is obtained, the total word frequency-inverse document frequency of each keyword in all papers is calculated. The calculation formula is as follows:

[0034]

[0035] K A (i) Sort by order, taking the top 10% of the keywords as the core words of the paper, and the number of core words Z as the number of clusters.

[0036] In a preferred embodiment of the present invention, the number of categories is set, and the data in multiple categories are subjected to keyword clustering analysis based on the number of categories to obtain clustering results, which specifically include:

[0037] Classify each keyword into a category, and each category contains only one keyword;

[0038] Calculate the distance between classes. By calculating the distance between the word frequency and inverse document frequency vector space of each class, find the two closest classes and merge them into one class, and get a new class with one less class than the original one.

[0039] Recalculate the word frequency-inverse document word frequency vector space distance between each pair in the new class, find the two closest classes and merge them into one class, and repeat this cycle until the required number of classes is reached, completing the hierarchical clustering of the paper keywords.

[0040] Compared with the prior art, the beneficial technical effects obtained by the present invention are:

[0041] Through big data technology, all keywords of publicly published papers in a certain discipline are retrieved, and all keywords are processed by word frequency inverse document word frequency, and thus a keyword word frequency-inverse document word frequency vector is constructed. The total word frequency-inverse document word frequency of each keyword in all papers is calculated, and they are sorted from large to small. The keywords that are ranked at the top with a certain percentage of the number of keywords are taken as the core words of the paper, and this core word is used as the name of the subdivided field, and the number of core words is used as the number of clusters to complete the hierarchical clustering of the paper keywords, and the discipline fields are divided according to the clustered keywords to meet the requirements of the division of different subdivided disciplines, so as to be more precise and accurate in the evaluation of disciplines and discipline talents, and to discover distinctive disciplines and talents in disciplines. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A flow chart of the subject field division method based on hierarchical clustering of paper keywords and semantics of the present invention is shown;

[0043] Figure 2 The flowchart of the keyword preprocessing method of the subject field division method based on hierarchical clustering of paper keywords and semantics of the present invention is shown;

[0044] Figure 3 The flowchart of the keyword classification method of the subject field division method based on hierarchical clustering of paper keywords and semantics of the present invention is shown. DETAILED DESCRIPTION

[0045] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0046] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0047] Please refer to Figure 1 , Figure 1 It is a flowchart of a method for dividing subject areas based on hierarchical clustering of paper keywords and semantics in some embodiments of the present application.

[0048] The first aspect of the present invention provides a method for dividing subject areas based on hierarchical clustering of paper keywords and semantics, comprising the following steps:

[0049] S1, based on big data technology, retrieve all keywords of publicly published papers in the specified discipline, pre-process all keywords, and obtain pre-processed keywords;

[0050] S2, analyzing semantics based on the preprocessed keywords to obtain semantic information, and classifying the keywords based on the semantic information to obtain multiple categories;

[0051] S3, setting the number of categories, performing keyword clustering analysis on the data in multiple categories based on the number of categories, and obtaining clustering results;

[0052] S4, classify the subject areas into categories based on the clustering results, obtain the subject area classification results, and transmit the subject area classification results to the terminal in real time.

[0053] Please refer to Figure 2 , Figure 2 This is a flowchart of a keyword preprocessing method for a subject field division method based on hierarchical clustering of paper keywords and semantics in some embodiments of the present application. According to an embodiment of the present invention, all keywords of publicly published papers in a set subject are retrieved based on big data technology, and all keywords are preprocessed to obtain preprocessed keywords, specifically including:

[0054] S101, determine the search subject, use big data technology to retrieve all publicly published papers in the subject, and obtain the keywords of each paper;

[0055] S102, identifying duplicate keywords in each paper, and merging the identified identical keywords to obtain pre-processed keywords.

[0056] Please refer to Figure 3 , Figure 3 This is a flow chart of a keyword classification method for a subject field division method based on hierarchical clustering of paper keywords and semantics in some embodiments of the present application. According to an embodiment of the present invention, semantics are analyzed based on pre-processed keywords to obtain semantic information, and keywords are classified based on the semantic information to obtain multiple categories, specifically including:

[0057] S201, using big data technology to retrieve the number of times each keyword appears in each paper, taking each keyword as a feature vector and the number of times it appears in each paper as a dimension value, to construct a keyword feature vector;

[0058] S202, performing word frequency-inverse document word frequency processing on the frequency of occurrence of keywords in each paper, and establishing a paper keyword word frequency-inverse document word frequency feature vector;

[0059] S203, calculating the total word frequency of each keyword in all papers minus the inverse document word frequency, sorting them from large to small, and taking the keywords in the sequence with the set percentage of the number of keywords as the core words of the paper;

[0060] S204, analyzing the semantics of the central word of the paper to obtain semantic information, using the semantic information and the central word of the paper as the name of the subdivided field, and using the number of central words as the number of clusters to obtain multiple categories.

[0061] According to an embodiment of the present invention, establishing a feature vector of the frequency of occurrence of keywords in a paper specifically includes:

[0062] Obtain data on M papers and N keywords, and retrieve the number of times each keyword appears in each paper based on big data technology;

[0063] Taking each keyword as a feature vector and the number of times it appears in each paper as the dimension value, the constructed keyword feature vector is:

[0064] Keyword K1=[k 11 ,k 12 ,......,k 1M ]

[0065] Keyword K2 = [k 21 ,k 22 ,......,k 2M ] ......

[0067] Keyword K N =[k N1 ,k N2 ,......,k NM ].

[0068] According to an embodiment of the present invention, a feature vector of paper keyword frequency-inverse document frequency is established, specifically including:

[0069] After constructing the feature vector of the frequency of keyword occurrence in the paper, the influence of common words on the frequency is eliminated, and the frequency of keyword occurrence in each paper is processed by word frequency-inverse document word frequency. The calculation formula is:

[0070]

[0071] In the formula, k ij is the number of times keyword i appears in article j, is the total number of times keyword i appears in all papers, log is the logarithm, D is the total number of papers in the subject, and D i The number of papers in which keyword i appears in the subject papers is used to establish the feature vector of keyword frequency-inverse document frequency:

[0072] Keyword Ks1=[S 11 , S 12 ,,......,S 1M ]

[0073] Keyword Ks2=[S 21 , S 22 ,,……,S 2M ] ......

[0075] Keyword Ks N =[S N1 , S N2 ,,......,S NM ].

[0076] According to an embodiment of the present invention, the method for determining the core words of a paper and the number of clusters is as follows:

[0077] After the paper keyword frequency-inverse document frequency feature vector is obtained, the total word frequency-inverse document frequency of each keyword in all papers is calculated. The calculation formula is as follows:

[0078]

[0079] K A (i) Sort by order, taking the top 10% of the keywords as the core words of the paper, and the number of core words Z as the number of clusters.

[0080] According to an embodiment of the present invention, the number of categories is set, and keyword clustering analysis is performed on data in multiple categories based on the number of categories to obtain clustering results, which specifically include:

[0081] Classify each keyword into a category, and each category contains only one keyword;

[0082] Calculate the distance between classes. By calculating the distance between the word frequency and inverse document frequency vector space of each class, find the two closest classes and merge them into one class, and get a new class with one less class than the original one.

[0083] Recalculate the word frequency-inverse document word frequency vector space distance between each pair in the new class, find the two closest classes and merge them into one class, and repeat this cycle until the required number of classes is reached, completing the hierarchical clustering of the paper keywords.

[0084] In summary, this application uses big data technology to retrieve all the keywords of publicly published papers in a certain discipline, performs word frequency inverse document word frequency processing on all keywords, and constructs a keyword word frequency-inverse document word frequency vector. Calculate the total word frequency-inverse document word frequency of each keyword in all papers, sort them from large to small, take the keywords that are ranked in front of a certain percentage of the number of keywords as the paper core words, use this core word as the name of the subdivided field, and use the number of core words as the number of clusters to complete the hierarchical clustering of paper keywords, divide the subject fields according to the clustered keywords, and realize the requirements of different subdivided fields of the discipline, so as to be more precise and accurate in the evaluation of disciplines and disciplinary talents, and discover distinctive disciplinary fields and talents in disciplinary fields.

[0085] The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0086] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0087] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0088] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for dividing subject areas based on hierarchical clustering of paper keywords and semantics, characterized in that: The steps include: Based on big data technology, all keywords of publicly published papers in a given discipline are retrieved, and all keywords are preprocessed to obtain preprocessed keywords; Analyze the semantics of the preprocessed keywords to obtain semantic information, and classify the keywords based on the semantic information to obtain multiple categories; Set the number of categories, perform keyword clustering analysis on the data in multiple categories based on the number of categories, and obtain clustering results; Based on the clustering results, the subject fields are classified to obtain the subject field division results, and the subject field division results are transmitted to the terminal in real time.

2. The subject field division method based on hierarchical clustering of paper keywords and semantics according to claim 1 is characterized in that: Based on big data technology, all keywords of publicly published papers in the specified discipline are retrieved and preprocessed to obtain the preprocessed keywords, including: Determine the search subject, use big data technology to retrieve all publicly published papers in the subject, and obtain the keywords of each paper; Duplicate keywords of each paper are identified, and the identified identical keywords are merged to obtain the preprocessed keywords.

3. The subject field division method based on hierarchical clustering of paper keywords and semantics according to claim 2 is characterized in that: Analyze the semantics of the preprocessed keywords to obtain semantic information, and classify the keywords based on the semantic information to obtain multiple categories, including: Apply big data technology to retrieve the number of times each keyword appears in each paper, take each keyword as a feature vector, and the number of times it appears in each paper as the dimension value to construct the keyword feature vector; Perform word frequency-inverse document word frequency processing on the frequency of keywords in each paper, and establish the paper keyword word frequency-inverse document word frequency feature vector; Calculate the total word frequency of each keyword in all papers - inverse document word frequency, sort from large to small, and take the keywords with the set percentage sequence of keywords as the core words of the paper; Analyze the semantics of the core words of the paper to obtain semantic information. Based on the semantic information and the core words of the paper, use them as the names of subdivided fields, and use the number of core words as the number of clusters to obtain multiple categories.

4. The subject field division method based on hierarchical clustering of paper keywords and semantics according to claim 3 is characterized in that: Establish the feature vector of the frequency of occurrence of keywords in the paper, including: Obtain data on M papers and N keywords, and retrieve the number of times each keyword appears in each paper based on big data technology; Taking each keyword as a feature vector and the number of times it appears in each paper as the dimension value, the constructed keyword feature vector is: Keyword K1=[k 11 , k 12 ,......,k 1M ] Keyword K2 = [k 21 , k 22 ,......,k 2M ] ...... Keyword K N =[k N1 , k N2 ,......,k NM ].

5. The subject field division method based on hierarchical clustering of paper keywords and semantics according to claim 4 is characterized in that: Establish the feature vector of keyword frequency-inverse document frequency of the paper, including: After constructing the feature vector of the frequency of keyword occurrence in the paper, the influence of common words on the frequency is eliminated, and the frequency of keyword occurrence in each paper is processed by word frequency-inverse document word frequency. The calculation formula is: In the formula, k ij is the number of times keyword i appears in article j, is the total number of times keyword i appears in all papers, log is the logarithm, D is the total number of papers in the subject, and D i The number of papers in which keyword i appears in the subject papers is used to establish the feature vector of keyword frequency-inverse document frequency: Keyword Ks1=[S 11 , S 12 ,,.......,S 1M ] Keyword Ks2=[S 21 , S 22 ,,......,S 2M ] ...... Keyword Ks N =[S N1 , S N2 ,,.......,S NM ].

6. The subject field division method based on hierarchical clustering of paper keywords and semantics according to claim 3 is characterized in that: The method for determining the core words and the number of clusters in the paper is as follows: After the paper keyword frequency-inverse document frequency feature vector is obtained, the total word frequency-inverse document frequency of each keyword in all papers is calculated. The calculation formula is as follows: To K A (i) Sort by order, taking the top 10% of the keywords as the core words of the paper, and the number of core words Z as the number of clusters.

7. The subject field division method based on hierarchical clustering of paper keywords and semantics according to claim 1 is characterized in that: Set the number of categories, perform keyword clustering analysis on the data in multiple categories based on the number of categories, and obtain clustering results, including: Classify each keyword into a category, and each category contains only one keyword; Calculate the distance between classes. By calculating the distance between the word frequency and inverse document frequency vector space of each class, find the two closest classes and merge them into one class, and get a new class with one less class than the original one. Recalculate the word frequency-inverse document word frequency vector space distance between each pair in the new class, find the two closest classes and merge them into one class, and repeat this cycle until the required number of classes is reached, completing the hierarchical clustering of the paper keywords.

Images