Scientific and technological innovation resource data application scene classification method

Abstract

The invention belongs to the field of data analysis, and provides a scientific and technological innovation resource data application scene classification method which comprises the steps of text acquisition, pre-cleaning and word segmentation, word segmentation screening, input vector acquisition and nearest neighbor classification model classification. The training process of the nearest neighbor classification model comprises the steps of resource library presetting, text preprocessing, word segmentation screening, input vector acquisition, training sample acquisition and nearest neighbor algorithm training. According to the method, application scene classification of scientific and technological innovation resource data is realized by utilizing word segmentation weight screening and a nearest neighbor algorithm, and integration of resource data and screening of scientific research resources are simplified.

Description

Technical Field

[0001] The present invention relates to the field of data analysis, and in particular to a method for classifying application scenarios of scientific and technological innovation resource data. Background Art

[0002] Scientific and technological innovation resource data usually includes text data such as patent data, technical reports and paper data. Scientific and technological innovation resource data can help researchers and developers provide technical support. However, when professionals search for resources, they often cannot directly obtain the most relevant resources, resulting in the inability to obtain appropriate text data or increasing the time taken to search. With the widespread application of machine learning, many classification models for text classification have emerged, which greatly improves the efficiency of text screening.

[0003] However, there is little research on classification models for application scenarios of scientific and technological innovation resource data in the existing technology. The existing models have low accuracy and efficiency in classification, which makes them unable to meet the needs of production and research. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a method for classifying application scenarios of scientific and technological innovation resource data, which utilizes word segmentation weight screening and nearest neighbor algorithm to realize application scenario classification of scientific and technological innovation resource data, thereby simplifying the integration of resource data and the screening of scientific research resources.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] A method for classifying application scenarios of scientific and technological innovation resource data, comprising:

[0007] Get the text to be classified;

[0008] Pre-cleaning and word segmentation are performed on the text to be classified to obtain a word set to be classified;

[0009] Using the improved TF-IDF to screen the word set to be classified, and obtaining a weight value to be classified and a core word set to be classified;

[0010] Combining the core word set to be classified and the weight value to be classified to obtain an input vector to be classified;

[0011] Inputting the input vector to be classified into a preset nearest neighbor classification model to obtain a target application scenario classification result; the target application scenario classification result includes: a main application field, an auxiliary application field, a target recommendation rate of the auxiliary application field, and a recommended research topic;

[0012] The training process of the nearest neighbor algorithm classification model includes:

[0013] Determine a preset resource library; the preset resource library includes: preset texts and research topics, application field labels, and auxiliary field recommendation rates corresponding to the preset texts;

[0014] De-noising, stop word removal and word segmentation are performed on each of the preset texts to obtain a word set to be trained corresponding to the preset text;

[0015] A threshold of 0.5 is set, and the weight value to be trained of each of the word sets to be trained is calculated using the improved TF-IDF, and the weight values ​​to be trained are screened to obtain the core word set to be trained corresponding to the preset text;

[0016] Combining the core word set to be trained corresponding to the preset text and the weight value to obtain an input vector to be trained corresponding to the preset text;

[0017] Combining the input vector to be trained corresponding to the preset text with the research topic, the application field label, and the auxiliary field recommendation rate to obtain a training sample;

[0018] The scikit-learn module is called to train the training samples using the nearest neighbor algorithm to obtain the trained nearest neighbor classification model.

[0019] Preferably, calling the scikit-learn module and using the nearest neighbor algorithm to train the training samples to obtain the trained nearest neighbor classification model includes:

[0020] Randomly divide the training samples into K non-overlapping sample subsets;

[0021] Each time, selecting one of the K non-overlapping sample subsets as a test set, and using the sample subsets of the K non-overlapping sample subsets except the test set as a training set;

[0022] The training set is trained using a nearest neighbor algorithm to obtain a temporary classification model;

[0023] Using the test set to test the temporary classification model, and obtaining a classification rate;

[0024] The average value of all the classification rates is calculated, and the average value is used as the training k value of the nearest neighbor classification model.

[0025] Preferably, the improved TF-IDF calculation formula includes: and p i =TF i *IDF i ;n 0is the total number of words in the word set to be trained; n i is the number of occurrences of the i-th word in the word set to be trained; TF i is the frequency of the i-th word in the word set to be trained; d i is the number of documents in the preset resource library that contain the i-th word segment in the word set to be trained; d 0 The total number of documents in the preset resource library; is the adjustment coefficient; IDF i is the inverse document frequency of the i-th word in the word set to be trained; p i is the weight value to be trained corresponding to the ith word in the word set to be trained.

[0026] Preferably, the resource library acquisition process includes:

[0027] Identify target scientific research websites;

[0028] Using a crawler to download text resources from the target scientific research website;

[0029] Adding a tag to the text resource;

[0030] Based on the target scientific research website, reading the citation data and publication date of the text resource;

[0031] According to the reference data and the publication date, the text resources are screened using preset standards to obtain text files that can be used for training, and the text files are classified and stored according to the labels.

[0032] Preferably, the tool for acquiring the input vector to be trained is Word2Vec.

[0033] Preferably, the training set is trained using a nearest neighbor algorithm to obtain a temporary classification model, including:

[0034] Using the input vector to be trained in the training set as input of the KNN classifier;

[0035] Taking the application domain label corresponding to the input vector to be trained as the output of the KNN classifier;

[0036] The KNN classifier is trained to obtain the temporary classification model.

[0037] Preferably, the voting parameter of the KNN classifier is the Manhattan distance; the calculation formula of the Manhattan distance is: d = ∑|x m,p -x m,p-1 |; d is the Manhattan distance; x m,p is the mth vector parameter of the pth input vector to be trained.

[0038] Preferably, the training platform of the KNN classifier is PyCharm.

[0039] The present invention discloses the following technical effects:

[0040] The present invention provides a method for classifying application scenarios of scientific and technological innovation resource data. Through improved TF-IDF, the problem of excessive weights of some word segments in big data is solved, and reasonable screening of word segments is achieved. Through input vectors of multi-parameter combinations, the problem of single classification model function is solved, and reliable recommendations for main fields, auxiliary fields and topics of texts are achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A schematic diagram of the classification process of application scenarios of scientific and technological innovation resource data provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The purpose of the present invention is to provide a method for classifying application scenarios of scientific and technological innovation resource data, which utilizes word segmentation weight screening and nearest neighbor algorithm to realize application scenario classification of scientific and technological innovation resource data, and simplifies the integration of resource data and the screening of scientific research resources.

[0045] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Figure 1 A schematic diagram of a classification process of application scenarios of scientific and technological innovation resource data provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a method for classifying application scenarios of scientific and technological innovation resource data, including:

[0047] Step 100: Obtain the text to be classified;

[0048] Step 200: pre-clean and segment the text to be classified to obtain a word set to be classified;

[0049] Step 300: Use the improved TF-IDF to screen the word set to be classified, and obtain the weight value to be classified and the core word set to be classified;

[0050] Step 400: combining the core word set to be classified and the weight value to be classified to obtain an input vector to be classified;

[0051] Step 500: Input the input vector to be classified into a preset nearest neighbor classification model to obtain the classification result of the target application scenario; the classification result of the target application scenario includes: the main application field, the auxiliary application field, the target recommendation rate of the auxiliary application field and the recommended research topic;

[0052] The training process of the nearest neighbor classification model includes:

[0053] Determine the preset resource library; the preset resource library includes: preset texts and the research topics, application field labels and auxiliary field recommendation rates corresponding to the preset texts;

[0054] De-noise, remove stop words, and segment each preset text to obtain a set of words to be trained corresponding to the preset text;

[0055] Set the threshold to 0.5, use the improved TF-IDF to calculate the weight value of each word set to be trained, and filter the weight value to be trained to obtain the core word set to be trained corresponding to the preset text; the threshold setting can be adjusted according to the analysis effect of the final training model;

[0056] The core word set to be trained corresponding to the preset text and the weight value are combined to obtain the input vector to be trained corresponding to the preset text;

[0057] The input vector to be trained corresponding to the preset text is combined with the research topic, application field label and auxiliary field recommendation rate to obtain a training sample;

[0058] Call the scikit-learn module and use the nearest neighbor algorithm to train the training samples to obtain a trained nearest neighbor classification model.

[0059] Specifically, the scikit-learn module is called to train the training samples using the nearest neighbor algorithm to obtain a trained nearest neighbor classification model, including:

[0060] Randomly divide the training samples into K non-overlapping sample subsets;

[0061] Each time, a sample subset is selected from the K non-overlapping sample subsets as the test set, and the sample subsets of the K non-overlapping sample subsets except the test set are used as the training set;

[0062] The training set is trained using the nearest neighbor algorithm to obtain a temporary classification model;

[0063] Use the test set to test the temporary classification model and obtain the classification rate;

[0064] Calculate the average of all classification rates and use the average as the training k value for the nearest neighbor classification model.

[0065] Furthermore, the improved TF-IDF calculation formula includes: and p i =TF i *IDF i ;n 0 is the total number of words in the word set to be trained; n i is the number of occurrences of the i-th word in the training word set; TF i is the frequency of the i-th word in the word set to be trained; d i is the number of documents in the preset resource library that contain the i-th word in the word set to be trained; d 0 The total number of documents in the preset resource library; is the adjustment coefficient; IDF i is the inverse document frequency of the i-th word in the training word set; p i is the weight value to be trained corresponding to the i-th word in the word set to be trained.

[0066] Specifically, the resource library acquisition process includes:

[0067] Identify target scientific research websites;

[0068] Use crawlers to download text resources from target scientific research websites;

[0069] Add tags to text resources;

[0070] Based on the target scientific research website, read the citation data and publication date of the text resources;

[0071] According to the citation data and publication date, the text resources are screened using preset standards to obtain text files that can be used for training, and the text files are classified and stored according to labels.

[0072] Optionally, the tool for obtaining the input vector to be trained is Word2Vec.

[0073] Furthermore, the training set is trained using the nearest neighbor algorithm to obtain a temporary classification model, including:

[0074] The input vector to be trained in the training set is used as the input of the KNN classifier;

[0075] The application domain label corresponding to the input vector to be trained is used as the output of the KNN classifier;

[0076] Train the KNN classifier to obtain a temporary classification model.

[0077] Preferably, the voting parameter of the KNN classifier is the Manhattan distance; the calculation formula of the Manhattan distance is: d = ∑|x m,p -x m,p-1 |; d is Manhattan distance; x m,p is the mth vector parameter of the pth input vector to be trained.

[0078] Optionally, the training platform of the KNN classifier is PyCharm.

[0079] Specifically, the main application fields and auxiliary application fields correspond to the major fields and sub-fields of scientific research and innovation texts. This embodiment divides the major fields into: natural sciences: physics, chemistry, earth sciences, astronomy, biology, mathematics; engineering technology: electronic engineering, information technology, materials science, mechanical engineering, construction engineering, chemical technology; medicine and health: medicine, pharmacy, biomedical engineering, agricultural medicine, public health; social sciences: economics, political science, sociology, psychology, cultural studies, history; art and design: visual arts, music and dance, film and television media, product design, architectural design; other categories: environmental science, ecology, energy science, climate change research, education research.

[0080] Optionally, the subdivisions can be personalized according to the needs. The following are some of the subdivisions selected in this embodiment: Physics: particle physics, atomic physics, condensed matter physics, optics, astrophysics, thermodynamics and statistical physics, mechanics, nuclear physics, etc.; Electronic Engineering: communication engineering, control engineering, power electronics and motor control, microelectronics, big data and artificial intelligence, radio frequency and millimeter wave technology, etc. If you want to obtain a classification model with richer functions and more accurate classification, you can increase the subdivision level. For example, microelectronics can be divided into semiconductor devices, integrated circuit design, VLSI design, radio frequency integrated circuits, digital signal processing, MEMS / NEMS, semiconductor processes, new semiconductor materials, etc., among which semiconductor devices can also be subdivided into transistors, optoelectronic devices, power semiconductor devices, sensors, optoelectronic device integrated circuits, radio frequency devices, low-power devices, etc. When combining the main field and the subdivision field, you can add multiple different field names at the same level and attach different recommendation rates to each field.

[0081] Preferably, since this classification model can only be used for text classification research, the pictures and formulas in the file are uniformly replaced by their exclusive names.

[0082] Specifically, the model training of this embodiment adopts the PyCharm platform, which is a commonly used python integrated development environment. Before model training, the database and scikit-learn library files are imported to build a basic training environment for the nearest neighbor algorithm.

[0083] Optionally, in order to reduce the overfitting of the nearest neighbor algorithm, the amount of data in each field must not be too low. Therefore, the number of files in the last segment of the present embodiment is not less than 100.

[0084] Specifically, in order to verify the performance and generalization ability of the model, verification is required after the model is built. This embodiment uses a confusion matrix to measure the accuracy of the classification model. The confusion matrix is ​​as follows:

[0085]

[0086] Among them, A 11 is the total number of samples that are actually in Class 1 and predicted to be in Class 1; A 21 is the total number of samples predicted as Class 1 but actually as Class 2; N is the total number of training samples. Therefore, the calculation formula for accuracy is:

[0087]

[0088] The accuracy threshold set in this embodiment is 90%. Only when the accuracy of the model is not less than 90% can it be output for subsequent analysis.

[0089] The beneficial effects of the present invention are as follows:

[0090] The present invention utilizes word segmentation weight screening and the nearest neighbor algorithm to achieve application scenario classification of scientific and technological innovation resource data, thereby simplifying the integration of resource data and the screening of scientific research resources.

[0091] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0092] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A method for classifying application scenarios of scientific and technological innovation resource data, characterized in that: include: Get the text to be classified; Pre-cleaning and word segmentation are performed on the text to be classified to obtain a word set to be classified; Using the improved TF-IDF to screen the word set to be classified, and obtaining a weight value to be classified and a core word set to be classified; Combining the core word set to be classified and the weight value to be classified to obtain an input vector to be classified; Inputting the input vector to be classified into a preset nearest neighbor classification model to obtain a classification result of the target application scenario; The target application scenario classification results include: the main application field, the auxiliary application field, the target recommendation rate of the auxiliary application field, and the recommended research topic; The training process of the nearest neighbor algorithm classification model includes: Determine a preset resource library; the preset resource library includes: preset texts and research topics, application field labels, and auxiliary field recommendation rates corresponding to the preset texts; De-noising, stop word removal and word segmentation are performed on each of the preset texts to obtain a word set to be trained corresponding to the preset text; A threshold of 0.5 is set, and the weight value to be trained of each of the word sets to be trained is calculated using the improved TF-IDF, and the weight values ​​to be trained are screened to obtain the core word set to be trained corresponding to the preset text; Combining the core word set to be trained corresponding to the preset text and the weight value to obtain an input vector to be trained corresponding to the preset text; Combining the input vector to be trained corresponding to the preset text with the research topic, the application field label, and the auxiliary field recommendation rate to obtain a training sample; The scikit-learn module is called to train the training samples using the nearest neighbor algorithm to obtain the trained nearest neighbor classification model.

2. According to claim 1, a method for classifying application scenarios of scientific and technological innovation resource data is characterized in that: Calling the scikit-learn module and using the nearest neighbor algorithm to train the training samples to obtain the trained nearest neighbor classification model includes: Randomly divide the training samples into K non-overlapping sample subsets; Each time, selecting one of the K non-overlapping sample subsets as a test set, and using the sample subsets of the K non-overlapping sample subsets except the test set as a training set; The training set is trained using a nearest neighbor algorithm to obtain a temporary classification model; Using the test set to test the temporary classification model, and obtaining a classification rate; The average value of all the classification rates is calculated, and the average value is used as the training k value of the nearest neighbor classification model.

3. According to claim 1, a method for classifying application scenarios of scientific and technological innovation resource data is characterized in that: The improved TF-IDF calculation formula includes: and p i =TF i *IDF i ; n0 is the total number of words in the word set to be trained; n i is the number of occurrences of the i-th word in the word set to be trained; TF i is the frequency of the i-th word in the word set to be trained; d i is the number of documents in the preset resource library that contain the i-th word segment in the word set to be trained; d0 is the total number of documents in the preset resource library; is the adjustment coefficient; IDF i is the inverse document frequency of the i-th word in the word set to be trained; p i is the weight value to be trained corresponding to the ith word in the word set to be trained.

4. According to claim 1, a method for classifying application scenarios of scientific and technological innovation resource data is characterized in that: The resource library acquisition process includes: Identify target scientific research websites; Using a crawler to download text resources from the target scientific research website; Adding a tag to the text resource; Based on the target scientific research website, reading the citation data and publication date of the text resource; According to the reference data and the publication date, the text resources are screened using preset standards to obtain text files that can be used for training, and the text files are classified and stored according to the labels.

5. According to claim 1, a method for classifying application scenarios of scientific and technological innovation resource data is characterized in that: The tool for obtaining the input vector to be trained is Word2Vec.

6. A method for classifying application scenarios of scientific and technological innovation resource data according to claim 2, characterized in that: The training set is trained using a nearest neighbor algorithm to obtain a temporary classification model, including: Using the input vector to be trained in the training set as input of the KNN classifier; Taking the application domain label corresponding to the input vector to be trained as the output of the KNN classifier; The KNN classifier is trained to obtain the temporary classification model.

7. A method for classifying application scenarios of scientific and technological innovation resource data according to claim 6, characterized in that: The voting parameter of the KNN classifier is the Manhattan distance; the calculation formula of the Manhattan distance is: d = ∑x m,p -x m,p-1 |; d is the Manhattan distance; x m,p is the mth vector parameter of the pth input vector to be trained.

8. A method for classifying application scenarios of scientific and technological innovation resource data according to claim 6, characterized in that: The training platform of the KNN classifier is PyCharm.

Images