Hierarchical clustering and directory automatic construction method based on data features of state grid documents

Abstract

The present application relates to a kind of based on the hierarchical clustering and directory automatic construction method of State Grid document data feature, comprising the following steps: step S1: obtain State Grid document data, and pre-process;Step S2: with TF-IDF algorithm, document name and document title are identified with the keyword table of category information;Step S3: according to keyword table identification result, word2vec tool is used to select Skip-gram model and generate keyword vector for the training of word embedding model;Step S4: in combination with document vectorization representation result, State Grid document hierarchical clustering is carried out using the condensed hierarchical clustering algorithm;Step S5: according to State Grid document hierarchical clustering result, according to the keyword in step S2 in the co-occurrence of the same category document set, set corresponding category label, complete State Grid document hierarchical directory construction.The present application effectively mines the document data features of State Grid field, realizes the automatic generation of State Grid document directory hierarchy, and improves the daily office efficiency of State Grid user.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of document management, and in particular to a hierarchical clustering and directory automatic construction method based on State Grid document data features. BACKGROUND

[0002] At present, there are massive black-box managed unstructured document data in the State Grid unstructured data management platform, and the total storage amount has reached 3PB, accounting for more than 70% of the total data amount. Black-box management only supports business systems to access and manage according to document IDs. Due to the lack of document category information, a visual document data directory that is convenient for online retrieval and batch sharing has not been formed, and it is difficult to provide sample data to typical application intelligent models, so there is an urgent need for a document directory automatic construction technology based on hierarchical clustering to release the value of massive document data of the State Grid.

[0003] However, with the explosive growth of the data scale of the State Grid and the increase in the complexity of the data field features, existing hierarchical clustering algorithms such as quasi-cluster center and natural neighbor are prone to ignore the field information existing in the clustering process when clustering large-scale field data, thereby easily leading to clustering interruption, low accuracy and other problems, so that directory construction often needs a lot of manual participation and is time-consuming and laborious.

[0004] It is necessary to combine the category features of the State Grid field document data, study a State Grid document data directory automatic construction method based on hierarchical clustering, mine key category feature information of the State Grid document, and automatically build an unstructured data directory and publish it to provide a "one-stop" visual, searchable and usable data service for State Grid users. SUMMARY

[0005] To solve the above problems, the purpose of the present application is to provide a hierarchical clustering and directory automatic construction method based on State Grid document data features, which effectively mines the document data features of the State Grid field and realizes the automatic generation of the hierarchical structure of the State Grid document directory, thereby improving the daily office efficiency of State Grid users.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] A hierarchical clustering and directory automatic construction method based on State Grid document data features comprises the following steps:

[0008] Step S1: Obtain State Grid document data and pre-process it;

[0009] Step S2: Identify the keyword table with category information of the document name and document title by using the TF-IDF algorithm;

[0010] Step S3: According to the keyword table recognition result, the word2vec tool is used to select the Skip-gram model for word embedding model training and generate keyword vector;

[0011] Step S4: Combined with the document vectorization representation result, the Agglomerative Clustering algorithm is used for hierarchical clustering of State Grid documents;

[0012] Step S5: According to the hierarchical clustering results of State Grid documents, according to the co-occurrence of keywords in step S2 in the same category document set, the corresponding category label is set, and the hierarchical directory construction of State Grid documents is completed, which is convenient for State Grid users to search and share the massive State Grid documents and improve the efficiency of daily office work.

[0013] Further, the preprocessing is as follows: the State Grid document data is segmented and State Grid named entity recognition is performed, the NLPIR segmentation system pynlpir component is used for text segmentation combined with the power grid entity name corpus, and the part-of-speech is labeled, the special entity names such as "equipment name, business name, organization name" in the State Grid document are labeled, and the stop word filtering processing is performed on the segmentation result, and the exclamation words, mood words, personal pronouns, and other words without specific document category meaning are deleted.

[0014] Further, step S2 is specifically:

[0015] First, the TF-IDF algorithm is used to identify keywords with category information for document name and document title, and the formula is as follows:

[0016] TF_IDF(t)=tf(t)*idf(t);

[0017] Wherein, TF is the word frequency, and IDF is the inverse document frequency;

[0018] Then, the TF-IDF algorithm is also used to identify keywords for document content, and finally the top n keywords of the document name and document title and the top m keywords of the document content are taken to form the keyword table of the document, and if the keywords are repeated, the keywords are combined.

[0019] Further, step S3 obtains the document vector by averaging the word vectors of the keywords:

[0020]

[0021] Wherein, V(d) represents a feature vector representation of a document, and the word vector of the i-th feature word c in the text is represented by v(c1), that is, the vectorization representation of the document is the average value of the sum of all keyword vectors.

[0022] Further, step S4 is specifically:

[0023] Step S41: Initially, each document point is taken as an independent cluster;

[0024] Step S42: Calculate the similarity between clusters by Euclidean distance. According to the mean linkage and cosine similarity, find the two closest clusters C i and C j；

[0025] For cluster C i containing N i sample points, its centroid is represented as:

[0026]

[0027] The distance between clusters C i and C j , based on their centroids, is defined as:

[0028]

[0029] where, is the centroid of cluster C j ;

[0030] Step S43: Merge clusters C i and C j into a new cluster C ij , and update the distance matrix;

[0031] Step S44: Repeat the above process until all documents are merged into one total cluster or reach the specified number of clusters k:

[0032] Step S45: Finally, form a hierarchical tree directory.

[0033] Further, step S5 is specifically:

[0034] After performing hierarchical clustering on the State Grid documents using the agglomerative hierarchical clustering algorithm, multiple categories (C1, C2, …, Ck) can be obtained, each category Ck containing several documents;

[0035] Count the keywords of all documents in category Ck and calculate the frequency of each keyword:

[0036]

[0037] where, represents the number of occurrences of keyword wi in document d;

[0038] Combine the TF-IDF weight to weight the keyword frequency:

[0039]

[0040] wherein, TF-IDF(w i ) is the TF-IDF value of the keyword w i ; FreqC k (w i ) is the total frequency of the keyword w i in the category C k ;

[0041] Sort the keywords according to Score Ck (w i ), and select the top N keywords with the highest scores as the category labels;

[0042] According to the hierarchical clustering results, organize the documents into a tree-shaped hierarchical directory structure by category, and apply the extracted category labels to the directory nodes. For the top-level category, select the keywords of all documents in the category for statistics, and extract high-frequency keywords as the root directory label. For each subcategory, repeat the above keyword extraction method to extract keywords based on the document set of the subcategory, and generate more fine-grained labels. Each document is classified into its corresponding clustering category and placed under the leaf node of the tree-shaped directory.

[0043] The present application has the following beneficial effects:

[0044] The present application effectively mines the document data features in the State Grid field, realizes the automatic generation of the State Grid document directory hierarchy, reduces the cost of manual correction, supports the convenient retrieval and sharing application of the massive document data of the State Grid, and improves the daily office efficiency of the users of the State Grid. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The present application has the following beneficial effects: DETAILED DESCRIPTION

[0046] The present application will be further described in detail below in combination with the drawings and specific embodiments:

[0047] Reference Figure 1 In the present embodiment, a hierarchical clustering and directory automatic construction method based on the document data features of the State Grid is provided, comprising the following steps:

[0048] Step S1: Obtain the document data of the State Grid and pre-process;

[0049] Step S2: Use the TF-IDF algorithm to identify the keyword table with category information for the document name and document title;

[0050] Step S3: According to the keyword table identification result, use the word2vec tool to select the Skip-gram model for word embedding model training and generate keyword vectors;

[0051] Step S4: combine the document vectorization representation results, and use an agglomerative clustering algorithm to perform hierarchical clustering of the state grid documents;

[0052] Step S5: according to the hierarchical clustering results of the state grid documents, set the corresponding category label according to the keywords in step S2 co-occurring in the same category document set, complete the hierarchical directory construction of the state grid documents, facilitate the state grid user to search and share the massive state grid documents, and improve the daily office efficiency.

[0053] In the embodiment, the preprocessing is specifically as follows: the state grid document data is subjected to word segmentation and state grid named entity recognition, the pynlpir component of the NLPIR word segmentation system is used for text word segmentation in combination with the power grid entity name corpus, part-of-speech tagging is performed, the special entity names such as “device name, business name, and organization name” in the state grid documents are labeled, and the word segmentation results are subjected to stop word filtering processing to delete exclamation words, mood words, personal pronouns, and the like.

[0054] In the embodiment, step S2 is specifically as follows:

[0055] Firstly, the TF-IDF algorithm is used to identify the keywords with category information for the document name and the document title, and the formula is as follows:

[0056] TF_IDF(t) = tf(t) * idf(t);

[0057] Wherein, TF is the word frequency, and IDF is the inverse document frequency.

[0058] Then, the TF-IDF algorithm is also used to identify the keywords for the document content, finally the top n keywords of the document name and the document title and the top m keywords of the document content are taken to form the keyword table of the document, and if the keywords are repeated, the keywords are combined.

[0059] In the embodiment, step S3 obtains the document vector by performing the average value operation on the word vectors of the keywords:

[0060]

[0061] Wherein, V(d) represents the feature vector representation of a document, and the word vector of the i-th feature keyword c in the text is represented by v(ci), that is, the vectorization representation of the document is the average value of the sum of all keyword vectors.

[0062] In the embodiment, step S4 is specifically as follows:

[0063] Step S41: initially take each document point as an independent cluster;

[0064] Step S42: Calculate the similarity between clusters by Euclidean distance i and C j ;

[0065] For cluster C i containing N i sample points, its centroid is represented as:

[0066]

[0067] The distance between clusters C i and C j , based on their centroids, is defined as:

[0068]

[0069] where, is the centroid of cluster C j ;

[0070] Step S43: Merge clusters C i and C j into a new cluster C ij , and update the distance matrix;

[0071] Step S44: Repeat the above process until all documents are merged into one total cluster or reach the specified number of clusters k:

[0072] Step S45: Finally form a hierarchical tree directory.

[0073] In this embodiment, step S5 is specifically:

[0074] After the hierarchical division of State Grid documents by the agglomerative hierarchical clustering algorithm, a plurality of categories (C1, C2, …, Ck) can be obtained, each category Ck containing a plurality of documents;

[0075] The keywords of all documents in category Ck are counted, and the frequency of each keyword is calculated:

[0076]

[0077] where, represents the number of occurrences of keyword wi in document d;

[0078] The keyword frequency is weighted by combining the TF-IDF weight:

[0079]

[0080] where, TF-IDF(w i ) is the keyword wi TF-IDF value of the keyword w in class C k (w i ) is the keyword w i Total frequency of occurrence in class C k

[0081] Sort keywords by Score Ck (w i ), and select top N keywords with highest scores as class labels

[0082] According to the hierarchical clustering result, organize documents into a tree-structured hierarchical directory structure by class, and apply the extracted class labels to the directory nodes. For the top-level class, select keywords from all documents in the class, and extract high-frequency keywords as root directory labels. For each sub-class, repeat the keyword extraction method to extract keywords based on the document set of the sub-class, and generate more granular labels. Place each document in its corresponding clustered class under the leaf node of the tree-shaped directory.

[0083] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) having computer-usable program code embodied therein.

[0084] The present application is described in reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in the flow or flows and / or blocks. Figure 1 The means for performing the functions specified in the flow or flows and / or blocks.

[0085] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in the flow or flows and / or blocks. Figure 1 ​the function specified in the one or more blocks.

[0086] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, so that the instructions executed on the computer or other programmable data processing devices provide processes for implementing the flow Figure 1 the flow or flows and / or blocks Figure 1 the steps of the function specified in the one or more blocks.

[0087] The above description is only the preferred embodiment of the present application, not other forms of the present application, any skilled in the art can use the above disclosed technical content to change or modify as equivalent embodiments of equivalent changes. But any simple modification, equivalent change and modification of the above embodiments without departing from the technical solution of the present application, according to the technical essence of the present application, still belongs to the protection scope of the technical solution of the present application.

Claims

1. A hierarchical clustering and directory automatic construction method based on State Grid document data features, characterized in that, The method comprises the following steps: Step S1: obtaining State Grid document data and preprocessing; Step S2: identifying keywords with category information from document names and document titles by using a TF-IDF algorithm; Step S3: training a word embedding model by using a Skip-gram model of a word2vec tool according to the keyword identification result, and generating a keyword vector; Step S4: performing hierarchical clustering of State Grid documents by using a condensed hierarchical clustering algorithm in combination with a document vectorization representation result; Step S5: setting corresponding category labels according to keywords in step S2 co-occurring in a same category document set according to the hierarchical clustering result of the State Grid documents, and completing hierarchical directory construction of the State Grid documents; The step S4 is specifically: Step S41: initially taking each document point as an independent cluster; Step S42: Calculate the similarity between clusters by Euclidean distance i and C j； For clusters C i comprising N i sample points, whose centroid is represented as: ； The distance between clusters C i and C j is defined based on their centroids as: ； ； wherein is a cluster C j of centroids; Step S43: merge clusters C i and C j into new cluster C ij and update distance matrix; Step S44: repeating the above process until all documents are merged into a total cluster or a specified number of clusters k is reached; Step S45: finally forming a hierarchical tree directory; The step S5 is specifically: After the hierarchical division of the State Grid documents by the agglomerative hierarchical clustering algorithm, a plurality of categories (C1, C2, …, C k ) are obtained, each category C k contains a plurality of documents; For Category C k The keyword statistics of all documents in Category C are calculated, and the frequency of each keyword is calculated: ； wherein, denotes the keyword w i number of occurrences in the document d; combining TF-IDF weights to weight keyword frequencies: ； where TF-IDF(w i ) is the TF-IDF value of keyword w i ; FreqC k (w i ) is the total occurrence frequency of keyword w i in category C k ; and according to , the keywords are sorted, and the top N keywords with the highest scores are selected as the category labels. and according to the hierarchical clustering result, organizing documents into a tree-shaped hierarchical directory structure according to categories, and applying extracted category labels to directory nodes, for a top-level category, selecting keywords of all documents in the category for statistics, extracting high-frequency keywords as root directory labels, for each subcategory, repeating step S5, extracting keywords based on the document set of the subcategory, and generating more fine-grained labels; each document is placed in its corresponding clustering category under the leaf node of the tree-shaped directory.

2. The hierarchical clustering and directory automatic construction method based on the features of the State Grid document data according to claim 1, characterized in that, The preprocessing is specifically as follows: performing word segmentation and State Grid named entity recognition on the State Grid document data, using the pynlpir component of the NLPIR word segmentation system to perform text word segmentation while performing part-of-speech tagging, labeling out proprietary entity names in the State Grid documents, including device names, business names, and organization names, and performing stop word filtering on the word segmentation result to delete words without specific document category meanings: interjection words, modal words, and personal pronouns. 3.The method of claim 1, wherein, The step S2 is specifically: First, keywords with category information are identified from document names and document titles by using a TF-IDF algorithm, and the formula is as follows: ； Wherein, TF is the word frequency, and IDF is the inverse document frequency; Then, keywords are identified from document content by using the TF-IDF algorithm, and finally, the first n keywords of the document name and the document title and m keywords of the document content are taken to form a keyword table of the document, and if repeated, the keywords are combined.

4. The method of claim 1, wherein the method is characterized by, The step S3 obtains a document vector by averaging the word vectors of the keywords: ； wherein, represents a feature vector representation of a document, the word vector of the i-th feature word c in the text is denoted by v(c i ), i.e. the vectorized representation of the document is the average of the sum of all keyword vectors.

Images