A data classification method based on deep learning and a method for establishing a graph

Through the data classification method based on deep learning, using keyword weights and article matching for classification, the problem of inefficiency of clustering algorithms in large-scale data processing is solved, and the rapid classification of massive data and the establishment of knowledge graphs is realized, which reduces the Storage and calculation costs.

CN113886587BActive Publication Date: 2024-11-15HANGZHOU FANEWS TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111176377.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-09
Publication Date
2024-11-15
Estimated Expiration
2041-10-09

AI Technical Summary

Technical Problem

Existing clustering algorithms are inefficient when processing large-scale industry article data, resulting in increased storage burden and high manual classification costs, making it difficult to adapt to the rapid growth of massive data.

Method used

Using a deep learning-based data classification method, classify keyword weights and article matching degrees, use the TextRank algorithm to extract core keywords, establish a weight table, iteratively replace low-match articles, form an article classification model, reduce storage requirements and improve calculations efficiency.

Benefits of technology

It realizes the rapid and accurate classification of massive data, reduces storage and computing costs, improves the speed of model optimization, and can quickly identify target industry articles and establish knowledge graphs to efficiently retrieve massive data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113886587B_ABST
    Figure CN113886587B_ABST
Patent Text Reader

Abstract

The present invention provides a data classification method based on deep learning and a method for establishing a graph, wherein the data classification method includes: extracting core keywords from basic articles, calculating weight values ​​of core keywords, establishing a first weight correspondence table, extracting keywords from each basic article, calculating the industry matching degree of each article according to the first weight correspondence table, obtaining a first matching degree threshold, iterating the basic articles according to the first matching degree; repeating the above steps according to the iterated articles to obtain a second weight correspondence and a second matching degree threshold; using the second matching degree threshold to determine whether the new article belongs to the target industry. The present invention uses basic articles to extract keywords and divides the keywords into title keywords and text keywords, assigns different adjustment factors, can more effectively calculate the industry matching degree, and then uses articles with higher matching degrees to replace and iterate more accurately, freeing up space for storing historical data and obtaining the optimal model more quickly.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to a data classification method based on deep learning and a method for establishing an industry knowledge graph based on the aforementioned data classification method, and specifically to a self-learning deep learning article data classification method. Background Art

[0002] Text clustering technology can be applied to industry data analysis. The system can collect massive amounts of article data from various fields through web crawlers every day, and use algorithms to effectively classify these articles, which can help users quickly understand current industry information and conduct further analysis and processing efficiently.

[0003] At present, for data aggregation and data model establishment in specific industries, clustering algorithms are generally used to cluster data, and then the clustered data is manually statistically classified. However, clustering algorithms need to save all historical document information, which will cause storage burden; and there are a large number of new texts in various industries every day. When there are more and more articles, there will be more historical document information, which will lead to lower analysis and calculation efficiency of clustering algorithms. Therefore, clustering algorithms are only applicable to scenarios with small data volumes. When the data volume is large, the efficiency will be low, and the cost of manual classification will also be increased. Summary of the invention

[0004] In order to solve the problem of low efficiency of clustering algorithms when clustering articles in the same industry, the present application provides an article classification method, which uses keyword weights and the matching degree between articles and models for classification.

[0005] A data classification method based on deep learning includes the following steps:

[0006] Acquire a number of basic articles, extract a number of core keywords from the basic articles, calculate weight values ​​of the core keywords, and establish a first weight correspondence table according to the core keywords and the weight values;

[0007] Extract the title keywords and the text keywords in each basic article, query the weight values ​​of the title keywords and the text keywords according to the first weight correspondence table, and calculate the industry matching degree of each basic article according to the weight values ​​of the title keywords and the text keywords;

[0008] Obtaining a first matching degree threshold according to the industry matching degree of each basic article;

[0009] Extract the title keywords and text keywords of the article to be matched, calculate the industry matching degree of the article to be matched, when the industry matching degree of the article to be matched is greater than the first matching degree threshold, the article to be matched replaces the basic article with the lowest industry matching degree among the basic articles, when the industry matching degrees of all basic articles are greater than the first matching degree threshold, stop iteration, and obtain an article classification model;

[0010] Repeat the above steps using the articles in the article classification model to obtain a second weight correspondence table, and obtain a second matching degree threshold according to the second weight correspondence table;

[0011] The article classification model is used to calculate the industry matching degree of the article to be calculated. When the industry matching degree of the article to be calculated is greater than the second matching degree threshold, it is determined that the article to be calculated belongs to the target industry.

[0012] Furthermore, the method for calculating the weight value of the core keyword is specifically as follows:

[0013]

[0014] wordw[i] refers to the weight of the ith keyword in the model, fq[i] refers to the frequency of the ith keyword in this basic article, fqm[i] refers to the frequency of the ith keyword in all basic articles, and k refers to the number of basic articles.

[0015] Furthermore, the calculation method for calculating the industry matching degree of each basic article is as follows:

[0016]

[0017] Titlew is the title adjustment factor, contentw is the text adjustment factor, title[i] refers to the weight value corresponding to the i-th keyword appearing in the title in the first weight correspondence table, content[i] refers to the weight value corresponding to the i-th keyword appearing in the text in the first weight correspondence table.

[0018] Furthermore, the algorithm used to extract a plurality of core keywords from the plurality of basic articles is the TextRank algorithm.

[0019] Furthermore, the method of obtaining several basic articles includes: using crawler technology to collect data, using ElasTicSearch cluster storage, and using Hanlp word segmenter to perform full-text retrieval.

[0020] Furthermore, the industry matching degree of the article to be calculated is calculated as follows:

[0021]

[0022] Titlew is the title adjustment factor, contentw is the text adjustment factor, title[i] refers to the weight value corresponding to the i-th keyword appearing in the title in the second weight correspondence table, content[i] refers to the weight value corresponding to the i-th keyword appearing in the title in the second weight correspondence table.

[0023] The present invention also provides a method for establishing a graph, which uses the deep learning-based data classification method described above to obtain article data of the target industry and establish a knowledge graph of the target industry based on the article data.

[0024] Furthermore, a method for establishing a target industry knowledge graph based on the article data is specifically as follows: sampling articles according to keyword relevance to obtain sampled articles, extracting several keywords from the sampled articles, and deriving derivatives of the keywords through mutual information entropy calculation, and establishing an industry knowledge graph based on the keywords and derivatives.

[0025] Furthermore, it also includes calculating weight values ​​of the keywords and derivatives, and sorting the keywords and derivatives according to the weight values.

[0026] The present invention also provides an article classification device, comprising a memory and a processor, wherein the memory is used to store a data processing program, and when the data processing program is read and executed by the processor, it executes the data classification method based on deep learning described above.

[0027] The beneficial effects of the present invention are:

[0028] The present invention uses basic articles to extract keywords and divides the keywords into title keywords and text keywords, assigns different adjustment factors, and can more effectively calculate the industry matching degree of a certain article and form an article classification model. Then, this article classification model is replaced and iterated more accurately with articles with higher matching degrees. It is not necessary to store all historical data, which releases the storage space of the system, and fewer articles can reduce the amount of calculation. In the process of optimizing the model, the optimal model can be obtained more quickly. Then, the target industry articles are quickly identified through the matching degree threshold. After quickly obtaining the latest articles in a certain industry, a knowledge graph is established based on the fresh articles, and massive data is retrieved quickly and efficiently, which helps all walks of life to quickly condense themes in massive data and obtain valuable information. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 It is a flowchart of the article classification method;

[0031] Figure 2 It is a schematic diagram of the method for establishing an industry knowledge graph. DETAILED DESCRIPTION

[0032] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the embodiments described below are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0033] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments.

[0034] In the description of the present application, it should be understood that the terms "upper", "lower", "top", "bottom", "inside", "outside", etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0035] Example 1

[0036] This application provides a data classification method based on deep learning. First, articles are randomly selected to build a model, and then other articles are used to make the model self-learn. The final article classification model is obtained through iteration, and the final article classification model is used to judge industry articles. Specifically, the following steps are included:

[0037] By using crawler technology to collect data, using ElasTic Search cluster storage, and using Hanlp word segmenter for full-text retrieval, we can obtain several basic articles in the same industry from the data of industry websites, use the TextRank algorithm to extract several core keywords from the basic articles, calculate the weight values ​​of the core keywords, and establish a first weight correspondence table based on the core keywords and weight values.

[0038] Taking the forestry industry as an example, we selected 5,000 articles from the China Forestry Information Network as basic articles. A total of 500 core keywords were extracted from all these basic articles, and the weight values ​​of these core keywords were calculated. The first weight correspondence table was established using the sequence number, the core keyword itself, and the weight value.

[0039] Extract the title keywords and text keywords in each basic article, query the weight value of the title keywords and the weight value of the text keywords according to the first weight correspondence table, and calculate the industry matching degree of each basic article according to the weight value of the title keywords and the weight value of the text keywords.

[0040] Obtaining a first matching degree threshold according to the industry matching degree of each basic article;

[0041] Extract the title keywords and text keywords of the article to be matched, calculate the industry matching degree of the article to be matched, and when the industry matching degree of the article to be matched is greater than the first matching degree threshold, the article to be matched replaces the basic article with the lowest industry matching degree among the basic articles. When the industry matching degrees of all basic articles are greater than the first matching degree threshold, stop the iteration and obtain the article classification model. This is the process of self-updating learning of the model, the purpose of which is to find more similar articles with higher matching degrees to form a model for subsequent detection of articles to be detected, so as to make the article classification more accurate.

[0042] The above steps are repeated using the articles in the article classification model to obtain a second weight correspondence table, and a second matching degree threshold is obtained according to the second weight correspondence table.

[0043] The article classification model is used to calculate the industry matching degree of the article to be calculated. When the industry matching degree of the article to be calculated is greater than the second matching degree threshold, it is determined that the article to be calculated belongs to the target industry.

[0044] The specific method for calculating the weight value of core keywords is:

[0045]

[0046] wordw[i] refers to the weight of the ith keyword in the model, fq[i] refers to the frequency of the ith keyword in this basic article, fqm[i] refers to the frequency of the ith keyword in all basic articles, and k refers to the number of basic articles.

[0047] The first weight correspondence table of 500 core keywords is calculated by the above formula, where fq[i] corresponds to wordw[i], for example:

[0048] Serial number Core keywords Weight 1 forest 985 2 woodland 658 3 grassland 780 4 …… ……

[0049] As shown in the above chart, the core pipe detection corresponding to wordw[1] is forest, and the weight value of forest is 985.

[0050] The specific calculation method for calculating the industry matching degree of each basic article is as follows:

[0051]

[0052] Titlew is the title adjustment factor, which is used to control the importance ratio of the title in the calculation and is generally set to 1;

[0053] Contentw text adjustment factor, used to control the importance ratio of the text content in the calculation, generally set to 3;

[0054] title[i] refers to the weight value of the ith keyword in the title in the first weight correspondence table. For example, if the title of the article is "Which vegetation in my country is dominated by forests or grasslands?", then the core keywords in the title include forests and grasslands. If the corresponding relationship is found in the first weight table, then title[1] is the weight value of forests, which is the weight value of wordw[1], 985;

[0055] content[i] refers to the weight value corresponding to the i-th keyword appearing in the title in the first weight correspondence table.

[0056] Through the above method, it is calculated that the industry matching degree of these 5000 basic articles is 8000-20000, so the first matching degree threshold can be set to a value between 8000 and 20000. In this embodiment, the first matching degree threshold is 12000.

[0057] Get another batch of articles to be matched, extract the title keywords and text keywords of the articles to be matched, and use the above industry matching calculation formula to calculate the industry matching. When the industry matching of this article is greater than the first matching threshold of 12000, replace this article with the one with the lowest industry matching among the basic articles, that is, replace the basic article with an industry matching of 8000. Use other articles to be matched to iterate in the above manner. Finally, when the industry matching of all articles in the model is greater than the first matching threshold of 12000, stop the iteration and obtain the final article classification model.

[0058] Treat the articles in the current article classification model as "basic articles", repeat the above steps of calculating weight values ​​and calculating matching, and finally obtain the second weight correspondence table and the second matching threshold, which are used to subsequently determine whether the new article belongs to the target industry. It should be noted that the second matching threshold can be the lowest industry matching among all articles in the updated article classification model, or a value between the lowest matching and the highest matching. The second matching threshold can be formulated according to needs.

[0059] When a new article appears, the industry matching degree of the new article is calculated and judged using the second matching degree threshold, specifically:

[0060]

[0061] Titlew is the title adjustment factor, which is used to control the importance ratio of the title in the calculation and is generally set to 1;

[0062] Contentw text adjustment factor, used to control the importance ratio of the text content in the calculation, generally set to 3;

[0063] title[i] refers to the weight value of the ith keyword in the title in the second weight correspondence table. For example, if the title of the article is "Which vegetation in my country is dominated by forests or grasslands?", then the core keywords in the title include forests and grasslands. If the corresponding relationship is found in the second weight table, then title[1] is the weight value of forests, which is the weight value of wordw[1], 985;

[0064] content[i] refers to the weight value corresponding to the i-th keyword appearing in the text in the second weight correspondence table.

[0065] When the industry matching degree of a new article is greater than the second matching threshold, the new article belongs to the target industry and can be used as a database for establishing subsequent industry knowledge graphs.

[0066] Example 2

[0067] This embodiment provides a method for establishing a graph, specifically a method for establishing a target industry knowledge graph, using the article classification method in Embodiment 1 to obtain article data of a certain industry, and establishing a knowledge graph of the target industry based on the article data. Specifically, the following steps are included:

[0068] Article sampling is performed according to keyword relevance to obtain sampled articles, several keywords in the sampled articles are extracted, and derivatives of the keywords are obtained through mutual information entropy calculation, weight values ​​of all keywords and derivatives are calculated, keywords and derivatives are sorted according to weight values, and a topological relationship is established between keywords and derivatives to form a network structure diagram, thereby obtaining an industry knowledge graph.

[0069] The specific method for calculating the weight value is:

[0070]

[0071] Among them, Weight is used to represent weight;

[0072] subsetFreq is used to indicate the frequency of the word in the highly relevant result set;

[0073] subsetSize is used to indicate the size of the result set;

[0074] superFreq is used to represent the word frequency of the entire database;

[0075] superSize is used to indicate the size of the entire database;

[0076] natureBoost is used to represent the weights of parts of speech, where nouns and verbs have greater weights than other parts of speech;

[0077] fieldBoost is used to represent field weights, where the weight of the title is greater than the weight of the content.

[0078] like Figure 2 As shown, when the weight of the keyword is greater, the circle represented by the word is larger, and the circle of the derived word is smaller than the circle of the keyword. The specific establishment method can refer to the specification part of the patent application with publication number "CN112100399A" and "CN112100330A".

[0079] The knowledge graph model is created by creating keywords and corresponding derived phrases, and all keywords and derived words are sorted from top to bottom according to their own weight order, so that users can intuitively query the derived phrases of related keywords, and check the weights of each derived word in the derived phrase, and then use the knowledge graph model to perform corresponding searches. In this way, it is possible to use the knowledge graph model to obtain node sets, and search based on the node sets, and then quickly and efficiently perform graph search and discovery corresponding to the node sets in massive data, and it is helpful for all walks of life to quickly condense topics in massive data and obtain valuable information.

[0080] Example 3

[0081] This embodiment provides an article classification device, including a memory and a processor, wherein the memory is used to store a data processing program, and when the data processing program is read and executed by the processor, the data classification method based on deep learning in the embodiment 1 of the right is executed.

[0082] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0083] The units may or may not be physically separated, and the components displayed as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0084] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0085] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the above-mentioned computer-readable medium of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above.

[0086] More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductor segments, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk-read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0087] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless segments, wire segments, optical cables, RF, etc., or any suitable combination of the above.

[0088] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

Claims

1. A data classification method based on deep learning, characterized in that: The following steps are involved: Acquire a number of basic articles, extract a number of core keywords from the basic articles, calculate weight values ​​of the core keywords, and establish a first weight correspondence table according to the core keywords and the weight values; Extract the title keywords and the text keywords in each basic article, query the weight values ​​of the title keywords and the text keywords according to the first weight correspondence table, and calculate the industry matching degree of each basic article according to the weight values ​​of the title keywords and the text keywords; Obtaining a first matching degree threshold according to the industry matching degree of each basic article; Extract the title keywords and text keywords of the article to be matched, calculate the industry matching degree of the article to be matched, when the industry matching degree of the article to be matched is greater than the first matching degree threshold, the article to be matched replaces the basic article with the lowest industry matching degree among the basic articles, when the industry matching degrees of all basic articles are greater than the first matching degree threshold, stop iteration, and obtain an article classification model; Repeat the above steps using the articles in the article classification model to obtain a second weight correspondence table, and obtain a second matching degree threshold according to the second weight correspondence table; The article classification model is used to calculate the industry matching degree of the article to be calculated. When the industry matching degree of the article to be calculated is greater than the second matching degree threshold, it is determined that the article to be calculated belongs to the target industry.

2. According to the data classification method based on deep learning in claim 1, the method for calculating the weight value of the core keyword is specifically as follows: wordw[i] refers to the weight of the ith keyword in the model, fq[i] refers to the frequency of the ith keyword in this basic article, fqm[i] refers to the frequency of the ith keyword in all basic articles, and k refers to the number of basic articles.

3. According to the data classification method based on deep learning in claim 2, the calculation method of the industry matching degree of each basic article is specifically as follows: Titlew is the title adjustment factor, contentw is the text adjustment factor, title[i] refers to the weight value corresponding to the i-th keyword appearing in the title in the first weight correspondence table, content[i] refers to the weight value corresponding to the i-th keyword appearing in the title in the first weight correspondence table.

4. According to the deep learning-based data classification method of claim 1, the algorithm used to extract a plurality of core keywords from a plurality of the basic articles is the TextRank algorithm.

5. According to the data classification method based on deep learning in claim 1, the method for obtaining a plurality of basic articles comprises: Use crawler technology to collect data, use ElasTic Search cluster storage, and use Hanlp word segmenter for full-text retrieval.

6. According to the data classification method based on deep learning in claim 2, the industry matching degree of the article to be calculated is calculated as follows: Titlew is the title adjustment factor, contentw is the text adjustment factor, title[i] refers to the weight value corresponding to the i-th keyword appearing in the title in the second weight correspondence table, content[i] refers to the weight value corresponding to the i-th keyword appearing in the text in the second weight correspondence table.

7. A method for establishing a map, characterized in that: Use the deep learning-based data classification method described in any one of claims 1 to 6 to obtain article data of the target industry, and establish a knowledge graph of the target industry based on the article data.

8. According to the method for establishing a graph according to claim 7, a method for establishing a target industry knowledge graph based on the article data is specifically as follows: sampling articles according to keyword relevance to obtain sampled articles, extracting several keywords from the sampled articles, and calculating the derivatives of the keywords through mutual information entropy, and establishing an industry knowledge graph based on the keywords and derivatives.

9. The method for establishing a graph according to claim 8 further includes calculating weight values ​​of the keywords and derivatives, and sorting the keywords and derivatives according to the weight values.

10. An article classification device, characterized in that: It includes a memory and a processor, wherein the memory is used to store a data processing program, and when the data processing program is read and executed by the processor, it executes the deep learning-based data classification method described in any one of claims 1 to 6.