A technology innovation resource data application scenario classification method

Abstract

The application belongs to the field of data analysis, and provides a scientific and technological innovation resource data application scene classification method, which comprises the following steps: text acquisition, pre-cleaning and word segmentation, word segmentation screening, input vector acquisition and nearest neighbor classification model classification; the training process of the nearest neighbor classification model comprises the following steps: presetting a resource library, text preprocessing, word segmentation screening, input vector acquisition, training sample acquisition and nearest neighbor algorithm training. The application realizes the application scene classification of scientific and technological innovation resource data by using word segmentation weight screening and the nearest neighbor algorithm, and simplifies the integration of resource data and the screening of scientific research resources.

Description

Technical Field

[0001] This invention relates to the field of data analysis, and in particular to a method for classifying application scenarios of scientific and technological innovation resource data. Background Technology

[0002] Technological innovation resource data typically includes textual data such as patent data, technical reports, and academic papers. This data can provide technical support to researchers and developers. However, when professionals search for resources, they often cannot directly access the most relevant resources, leading to either an inability to obtain suitable textual data or increased search time. With the widespread application of machine learning, many classification models for text have emerged, significantly improving the efficiency of text filtering.

[0003] However, there is a lack of research on classification models for application scenarios of scientific and technological innovation resource data. Existing models have low accuracy and efficiency in classification, which makes them unable to meet the needs of production and research. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, the purpose of this invention is to provide a method for classifying application scenarios of scientific and technological innovation resource data. This method utilizes word segmentation weighting and nearest neighbor algorithms to classify application scenarios of scientific and technological innovation resource data, thereby simplifying the integration of resource data and the screening of scientific research resources.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A method for classifying application scenarios of science and technology innovation resource data includes:

[0007] Get the text to be categorized;

[0008] The text to be classified is pre-cleaned and segmented to obtain a set of words to be classified;

[0009] The improved TF-IDF was used to filter the word set to be classified, and the weight values ​​and core word set to be classified were obtained.

[0010] The core word set to be classified and the weight values ​​to be classified are combined to obtain the input vector to be classified.

[0011] The input vector to be classified is input into a preset nearest neighbor classification model to obtain the target application scenario classification result; the target application scenario classification result includes: main application domain, auxiliary application domain, target recommendation rate of the auxiliary application domain, and recommended research topic;

[0012] The training process of the nearest neighbor classification model includes:

[0013] Determine a preset resource library; the preset resource library includes: preset texts and the corresponding research topics, application domain tags, and auxiliary domain recommendation rates for the preset texts;

[0014] Each preset text is denoised, stop words are removed, and words are segmented to obtain a set of words to be trained corresponding to the preset text;

[0015] Set a threshold of 0.5, use the improved TF-IDF to calculate the training weight value of each training word set, and filter the training weight values ​​to obtain the training core word set corresponding to the preset text;

[0016] The training core word set and the training weight value corresponding to the preset text are combined to obtain the training input vector corresponding to the preset text.

[0017] The training sample is obtained by combining the input vector to be trained corresponding to the preset text, the research topic, the application domain label, and the auxiliary domain recommendation rate.

[0018] The scikit-learn module is called to train the training samples using the nearest neighbor algorithm, resulting in the trained nearest neighbor classification model.

[0019] Preferably, the scikit-learn module is called to train the training samples using the nearest neighbor algorithm to obtain the trained nearest neighbor classification model, including:

[0020] The training samples are randomly divided into K disjoint subsets;

[0021] Each time, select one of the K disjoint sample subsets as the test set, and use the sample subsets from the K disjoint sample subsets excluding the test set as the training set;

[0022] The training set is trained using the nearest neighbor algorithm to obtain a temporary classification model;

[0023] The temporary classification model is tested using the test set to obtain the classification rate;

[0024] Calculate the average of all the classification rates, and use the average as the training k value of the nearest neighbor classification model.

[0025] Preferably, the improved TF-IDF calculation formula includes: , , as well as ; The total number of words in the word set to be trained; The first word in the set of words to be trained The number of times each word appears; The first word in the set of words to be trained The word frequency of each segment; To include the first word in the training word set within the preset resource library The number of documents segmented into words; The total number of documents in the preset resource library; This is the adjustment coefficient; The first word in the set of words to be trained Inverse document frequency of each segment; The first word in the set of words to be trained The training weight values ​​corresponding to each word segmentation.

[0026] Preferably, the process of acquiring the resource library includes:

[0027] Identify target research websites;

[0028] The text resources of the target scientific research website were downloaded using a web crawler.

[0029] Add tags to the text resources;

[0030] Based on the target research website, retrieve the citation data and publication date of the text resource;

[0031] Based on the cited data and the publication date, the text resources are filtered using preset criteria to obtain text files that can be used for training, and the text files are classified and stored according to the tags.

[0032] Preferably, the tool for obtaining the input vector to be trained is Word2Vec.

[0033] Preferably, the training set is trained using the nearest neighbor algorithm to obtain a temporary classification model, including:

[0034] The input vector to be trained in the training set is used as the input to the KNN classifier;

[0035] The application domain label corresponding to the input vector to be trained is used as the output of the KNN classifier;

[0036] The KNN classifier is trained to obtain the temporary classification model.

[0037] Preferably, the voting parameter of the KNN classifier is the Manhattan distance; the formula for calculating the Manhattan distance is: ; The Manhattan distance; For the first The first of the training input vectors One vector parameter.

[0038] Preferably, the training platform for the KNN classifier is PyCharm.

[0039] The present invention discloses the following technical effects:

[0040] This invention provides a classification method for application scenarios of scientific and technological innovation resource data. By using an improved TF-IDF, it solves the problem of excessive weighting of some word segments in the case of big data, and realizes reasonable filtering of word segments. By using input vectors with multiple parameters, it solves the problem of the single function of the classification model, and realizes reliable recommendation of the main domain, auxiliary domain and topic of the text. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram illustrating the classification process of science and technology innovation resource data application scenarios provided in this embodiment of the invention. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] The purpose of this invention is to provide a method for classifying application scenarios of scientific and technological innovation resource data. This method utilizes word segmentation weighting and nearest neighbor algorithms to classify application scenarios of scientific and technological innovation resource data, thereby simplifying the integration of resource data and the screening of scientific research resources.

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0046] Figure 1 This is a schematic diagram of the classification process for application scenarios of scientific and technological innovation resource data provided in an embodiment of the present invention, such as... Figure 1 As shown, this invention provides a method for classifying application scenarios of science and technology innovation resource data, including:

[0047] Step 100: Obtain the text to be classified;

[0048] Step 200: Pre-clean and segment the text to be classified to obtain a set of words to be classified;

[0049] Step 300: Use the improved TF-IDF to filter the word set to be classified, and obtain the weight values ​​to be classified and the core word set to be classified;

[0050] Step 400: Combine the core word set to be classified and the weight values ​​to be classified to obtain the input vector to be classified;

[0051] Step 500: Input the input vector to be classified into the preset nearest neighbor classification model to obtain the target application scenario classification result; the target application scenario classification result includes: main application domain, auxiliary application domain, target recommendation rate of auxiliary application domain and recommended research topic;

[0052] The training process for the nearest neighbor classification model includes:

[0053] Determine the preset resource library; the preset resource library includes: preset texts and their corresponding research topics, application field tags, and auxiliary field recommendation rates;

[0054] For each preset text, noise reduction, stop word removal, and word segmentation are performed to obtain the training word set corresponding to the preset text;

[0055] Set a threshold of 0.5, use the improved TF-IDF to calculate the training weights for each training word set, and filter the training weights to obtain the core word set corresponding to the preset text; the threshold setting can be adjusted according to the analysis effect of the final training model.

[0056] The training core word set and training weight values ​​corresponding to the preset text are combined to obtain the training input vector corresponding to the preset text.

[0057] The training samples are obtained by combining the input vector to be trained corresponding to the preset text with the research topic, application domain labels and auxiliary domain recommendation rates;

[0058] The scikit-learn module is called to train the training samples using the nearest neighbor algorithm, resulting in a trained nearest neighbor classification model.

[0059] Specifically, the scikit-learn module is called to train the training samples using the nearest neighbor algorithm, resulting in a trained nearest neighbor classification model, including:

[0060] The training samples are randomly divided into K disjoint subsets;

[0061] Each time, select one subset of samples from the K disjoint sample subsets as the test set, and use the subset of samples from the K disjoint sample subsets excluding the test set as the training set;

[0062] A temporary classification model is obtained by training the training set using the nearest neighbor algorithm;

[0063] The temporary classification model was tested using a test set to obtain the classification accuracy.

[0064] Calculate the average of all classification rates and use the average as the training k value for the nearest neighbor classification model.

[0065] Furthermore, the improved formula for calculating TF-IDF includes: , , as well as ; The total number of words in the vocabulary set to be trained; The first word in the word set to be trained The number of times each word appears; For the first word in the word set to be trained The word frequency of each segment; To include the first word in the training word set within the preset resource library. The number of documents segmented into words; The total number of documents in the preset resource library; This is the adjustment coefficient; For the first word in the word set to be trained Inverse document frequency of each segment; For the first word in the word set to be trained The training weight values ​​corresponding to each word segmentation.

[0066] Specifically, the process of acquiring the resource repository includes:

[0067] Identify target research websites;

[0068] Use a web crawler to download text resources from the target research website;

[0069] Add tags to text resources;

[0070] Based on the target scientific research website, retrieve the citation data and publication date of the text resources;

[0071] Based on citation data and publication dates, text resources are filtered using preset criteria to obtain text files that can be used for training, and the text files are then classified and stored according to tags.

[0072] Optionally, the tool for obtaining the input vector to be trained is Word2Vec.

[0073] Furthermore, the nearest neighbor algorithm is used to train the training set to obtain a temporary classification model, including:

[0074] Use the input vector to be trained in the training set as the input to the KNN classifier;

[0075] The application domain label corresponding to the input vector to be trained is used as the output of the KNN classifier;

[0076] The KNN classifier is trained to obtain a temporary classification model.

[0077] Preferably, the voting parameter of the KNN classifier is the Manhattan distance; the formula for calculating the Manhattan distance is: ; Distance to Manhattan; For the first The first training input vector is the... One vector parameter.

[0078] Optionally, the training platform for the KNN classifier is PyCharm.

[0079] Specifically, the main application areas and auxiliary application areas correspond to the major categories and sub-categories of scientific research and innovation texts. This embodiment categorizes the major categories as follows: Natural Sciences: Physics, Chemistry, Earth Sciences, Astronomy, Biology, Mathematics; Engineering and Technology: Electronic Engineering, Information Technology, Materials Science, Mechanical Engineering, Architectural Engineering, Chemical Technology; Medicine and Health: Medicine, Pharmacy, Biomedical Engineering, Agricultural Medicine, Public Health; Social Sciences: Economics, Political Science, Sociology, Psychology, Cultural Studies, History; Art and Design: Visual Arts, Music and Dance, Film and Media, Product Design, Architectural Design; Others: Environmental Science, Ecology, Energy Science, Climate Change Research, Educational Research.

[0080] Optionally, the sub-fields can be customized according to needs. The following are some sub-fields selected in this embodiment: Physics: Particle Physics, Atomic Physics, Condensed Matter Physics, Optics, Astrophysics, Thermodynamics and Statistical Physics, Mechanics, Nuclear Physics, etc.; Electronic Engineering: Communication Engineering, Control Engineering, Power Electronics and Motor Control, Microelectronics, Big Data and Artificial Intelligence, RF and Millimeter Wave Technology, etc. To obtain a more feature-rich and precise classification model, more sub-levels can be added. For example, microelectronics can be divided into semiconductor devices, integrated circuit design, VLSI design, RF integrated circuits, digital signal processing, MEMS / NEMS, semiconductor processes, new semiconductor materials, etc. Semiconductor devices can be further subdivided into transistors, optoelectronic devices, power semiconductor devices, sensors, optoelectronic device integrated circuits, RF devices, low-power devices, etc. When combining the main field and sub-fields, multiple different field names at the same level can be added, and different recommendation rates can be assigned to each field.

[0081] Preferably, since this classification model can only be used for text classification research, the images and formulas in the file are uniformly replaced with their specific names.

[0082] Specifically, this embodiment uses the PyCharm platform for model training, which is a commonly used Python integrated development environment. Before model training, the database and scikit-learn library files are imported to build a basic training environment for the nearest neighbor algorithm.

[0083] Optionally, in order to reduce the overfitting of the nearest neighbor algorithm, the amount of data in each domain should not be too low. Therefore, in this embodiment, the number of files in the last subdivision domain is not less than 100.

[0084] Specifically, to verify the model's performance and generalization ability, validation is required after the model is built. This embodiment uses a confusion matrix to assess the accuracy of the classification model. The confusion matrix is ​​as follows:

[0085]

[0086] in, This represents the total number of samples that were actually classified as Class 1 but were predicted to be Class 1. The actual number of samples predicted to be of type 2 is the total number of samples predicted to be of type 1. Let be the total number of training samples. Therefore, the formula for calculating accuracy is:

[0087]

[0088] In this embodiment, the accuracy threshold is set to 90%. Only when the model's accuracy is not lower than 90% can it be output for subsequent analysis.

[0089] The beneficial effects of this invention are as follows:

[0090] This invention utilizes word segmentation weighting and nearest neighbor algorithms to classify application scenarios of science and technology innovation resource data, simplifying the integration of resource data and the screening of scientific research resources.

[0091] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0092] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for classifying application scenarios of science and technology innovation resource data, characterized in that, include: Get the text to be categorized; The text to be classified is pre-cleaned and segmented to obtain a set of words to be classified; The improved TF-IDF was used to filter the word set to be classified, and the weight values ​​and core word set to be classified were obtained. The core word set to be classified and the weight values ​​to be classified are combined to obtain the input vector to be classified. The input vector to be classified is input into a preset nearest neighbor classification model to obtain the classification result of the target application scenario; The target application scenario classification results include: main application domain, auxiliary application domain, target recommendation rate of the auxiliary application domain, and recommended research topics; The training process of the nearest neighbor classification model includes: Determine a preset resource library; the preset resource library includes: preset texts and the corresponding research topics, application domain tags, and auxiliary domain recommendation rates for the preset texts; Each preset text is denoised, stop words are removed, and words are segmented to obtain a set of words to be trained corresponding to the preset text; A threshold of 0.5 is set, and the training weight value for each of the training word sets is calculated using an improved TF-IDF algorithm. The training weight values ​​are then used to filter and obtain the core word set corresponding to the preset text. The improved TF-IDF calculation formula includes: , , as well as ; The total number of words in the word set to be trained; The first word in the set of words to be trained The number of times each word appears; The first word in the set of words to be trained The word frequency of each segment; To include the first word in the training word set within the preset resource library The number of documents segmented into words; The total number of documents in the preset resource library; This is the adjustment coefficient; The first word in the set of words to be trained Inverse document frequency of each segment; The first word in the set of words to be trained The training weight values ​​corresponding to each word segmentation; The training core word set and the training weight value corresponding to the preset text are combined to obtain the training input vector corresponding to the preset text. The training sample is obtained by combining the input vector to be trained corresponding to the preset text, the research topic, the application domain label, and the auxiliary domain recommendation rate. The scikit-learn module is called to train the training samples using the nearest neighbor algorithm, resulting in the trained nearest neighbor classification model.

2. The method for classifying application scenarios of science and technology innovation resource data according to claim 1, characterized in that, The scikit-learn module is called to train the training samples using the nearest neighbor algorithm, resulting in the trained nearest neighbor classification model, including: The training samples are randomly divided into K disjoint subsets; Each time, select one of the K disjoint sample subsets as the test set, and use the sample subsets from the K disjoint sample subsets excluding the test set as the training set; The training set is trained using the nearest neighbor algorithm to obtain a temporary classification model; The temporary classification model is tested using the test set to obtain the classification rate; Calculate the average of all the classification rates, and use the average as the training k value of the nearest neighbor classification model.

3. The method for classifying application scenarios of science and technology innovation resource data according to claim 1, characterized in that, The process of acquiring the resource library includes: Identify target research websites; The text resources of the target scientific research website were downloaded using a web crawler. Add tags to the text resources; Based on the target research website, retrieve the citation data and publication date of the text resource; Based on the cited data and the publication date, the text resources are filtered using preset criteria to obtain text files that can be used for training, and the text files are classified and stored according to the tags.

4. The method for classifying application scenarios of science and technology innovation resource data according to claim 1, characterized in that, The tool used to obtain the input vector to be trained is Word2Vec.

5. The method for classifying application scenarios of science and technology innovation resource data according to claim 2, characterized in that, The training set is trained using the nearest neighbor algorithm to obtain a temporary classification model, including: The input vector to be trained in the training set is used as the input to the KNN classifier; The application domain label corresponding to the input vector to be trained is used as the output of the KNN classifier; The KNN classifier is trained to obtain the temporary classification model.

6. The method for classifying application scenarios of science and technology innovation resource data according to claim 5, characterized in that, The voting parameter of the KNN classifier is the Manhattan distance; the formula for calculating the Manhattan distance is: ; The Manhattan distance; For the first The first of the training input vectors One vector parameter.

7. The method for classifying application scenarios of science and technology innovation resource data according to claim 5, characterized in that, The training platform for the KNN classifier is PyCharm.

Images