Hierarchical clustering and directory automatic construction method based on state grid document data features

Abstract

The invention relates to a hierarchical clustering and directory automatic construction method based on state grid document data features, which comprises the following steps of: S1, acquiring state grid document data, and preprocessing the state grid document data; s2, performing keyword table identification with category information on the document name and the document title by using a TF-IDF algorithm; s3, according to a keyword table recognition result, adopting a word2vec tool to select a Skip-gram model to train a word embedding model and generate a keyword vector; s4, combining a document vectorization representation result, and adopting a condensation type hierarchical clustering algorithm to perform state grid document hierarchical clustering; and S5, according to a state grid document hierarchical clustering result and the keywords in the step S2 of co-occurrence of the same category of document sets, setting corresponding category labels, and completing state grid document hierarchical directory construction. According to the method, the document data characteristics in the state grid field are effectively mined, the state grid document directory hierarchical structure is automatically generated, and the daily office efficiency of state grid users is improved.

Description

Technical Field

[0001] The present invention relates to the field of document management, and in particular to a hierarchical clustering and directory automatic construction method based on state grid document data features. Background Art

[0002] At present, there is a massive amount of unstructured document data managed in a black-box manner in the State Grid's unstructured data management platform, with a total storage volume of 3PB, accounting for more than 70% of the total data volume. Black-box management only supports business systems to manage access based on document IDs. Due to the lack of document category information, a visual document data directory that is convenient for online retrieval and batch sharing has not been formed, making it difficult to provide sample data for typical application intelligent models. There is an urgent need for automatic document directory construction technology based on hierarchical clustering to release the value of the State Grid's massive document data.

[0003] However, with the explosive growth of the scale of State Grid data and the increasing complexity of data domain characteristics, existing hierarchical clustering algorithms, such as quasi-cluster center and natural neighbor algorithms, tend to ignore the domain information in the clustering process when clustering large-scale domain data, which can easily lead to clustering interruptions, low accuracy and other problems. As a result, directory construction often requires a lot of manual participation, which is time-consuming and labor-intensive.

[0004] It is necessary to combine the characteristics of document data categories in the State Grid field, study the automatic construction method of the State Grid document data directory based on hierarchical clustering, mine the key classification feature information of the State Grid documents, automatically build and publish unstructured data directories, and provide "one-stop" visible, traceable, and usable data services for State Grid users. Summary of the invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide a hierarchical clustering and directory automatic construction method based on the characteristics of the State Grid document data, effectively mine the document data characteristics in the field of the State Grid, realize the automatic generation of the hierarchical structure of the State Grid document directory, and improve the daily office efficiency of the State Grid users.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A hierarchical clustering and directory automatic construction method based on the characteristics of state-owned grid document data includes the following steps:

[0008] Step S1: Obtain the State Grid document data and pre-process it;

[0009] Step S2: Using the TF-IDF algorithm to identify the keyword table with category information for the document name and the document title;

[0010] Step S3: According to the recognition results of the keyword table, the word2vec tool is used to select the Skip-gram model to train the word embedding model and generate keyword vectors;

[0011] Step S4: Based on the document vectorization results, an agglomerative hierarchical clustering algorithm is used to perform hierarchical clustering of the State Grid documents;

[0012] Step S5: According to the hierarchical clustering results of the State Grid documents, according to the keywords in step S2 that co-occur in the same category document set, set the corresponding category labels to complete the construction of the State Grid document hierarchical directory, so as to facilitate the State Grid users to retrieve and share the State Grid's massive documents and improve daily office efficiency.

[0013] Further, the preprocessing is as follows: word segmentation and named entity recognition of the State Grid document data are performed, and the pynlpir component of the NLPIR word segmentation system is used to perform text word segmentation in combination with the power grid entity name corpus, and part-of-speech tagging is performed at the same time, marking the proprietary entity names in the State Grid documents, such as "equipment name, business name, organization name", etc., and the stop word filtering processing is performed on the word segmentation results to delete interjections, modal particles, personal pronouns, and other words without specific document category meanings.

[0014] Furthermore, step S2 is specifically as follows:

[0015] First, the TF-IDF algorithm is used to identify keywords with category information in the document name and document title. The formula is as follows:

[0016] TF_IDF(t)=tf(t)*idf(t);

[0017] Among them, TF is term frequency and IDF is inverse document frequency;

[0018] Then, the TF-IDF algorithm is also used to identify keywords in the document content. Finally, the first n keywords in the document name and title of the document are taken, and the m keywords in the document content are taken to form the keyword table of the document. If repeated, keyword merging is performed.

[0019] Furthermore, step S3 obtains the document vector by averaging the word vectors of the key words:

[0020]

[0021] Among them, V(d) represents the feature vector representation of a document, and the word vector of the i-th feature word c in the text is represented by v(c1), that is, the vectorized representation of the document is the average of the sum of all keyword vectors.

[0022] Furthermore, step S4 is specifically as follows:

[0023] Step S41: initially treating each document point as an independent cluster;

[0024] Step S42: Calculate the similarity between clusters by Euclidean distance. According to mean link and cosine similarity, find the two clusters C with the closest distance. i and C j；

[0025] For cluster C i Contains N i sample points, whose centroid is expressed as:

[0026]

[0027] Cluster C i and C j The distance between , based on their centroids, is defined as:

[0028]

[0029] in, Cluster C j The centroid of

[0030] Step S43: Merge cluster C i and C j Become a new cluster C ij , and update the distance matrix;

[0031] Step S44: Repeat the above process until all documents are merged into one total cluster or the specified number of clusters k is reached:

[0032] Step S45: Finally, a hierarchical tree directory is formed.

[0033] Furthermore, step S5 is specifically as follows:

[0034] After the State Grid documents are hierarchically divided by the agglomerative hierarchical clustering algorithm, multiple categories (C1, C2, …, Ck) can be obtained, and each category Ck contains several documents;

[0035] Count the keywords of all documents in category Ck and calculate the frequency of occurrence of each keyword:

[0036]

[0037] in, Indicates the number of occurrences of keyword wi in document d;

[0038] Combine TF-IDF weights to weight keyword frequency:

[0039]

[0040] Among them, TF-IDF(w i ) is the keyword w i TF-IDF value of FreqC k (w i ) is the keyword w i In Category C k The total frequency of occurrence in ;

[0041] According to Score Ck (w i ) Sort the keywords and select the top N keywords with the highest scores as category labels;

[0042] According to the hierarchical clustering results, the documents are organized into a tree-like hierarchical directory structure by category, and the extracted category labels are applied to the directory nodes. For the top-level category, the keywords of all documents in the category are selected for statistics, and high-frequency keywords are extracted as root directory labels. For each subcategory, the above keyword extraction method is repeated, and keywords are extracted based on the document set of the subcategory to generate more fine-grained labels; each document is classified into its corresponding clustering category and placed under the leaf node of the tree directory.

[0043] The present invention has the following beneficial effects:

[0044] The present invention effectively mines the document data characteristics in the State Grid field, realizes the automatic generation of the State Grid document directory hierarchy, reduces the cost of manual participation in correction, supports the convenient retrieval and sharing application of the State Grid's massive document data, and improves the daily office efficiency of State Grid users. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] refer to Figure 1 In this embodiment, a hierarchical clustering and directory automatic construction method based on the characteristics of the State Grid document data is provided, comprising the following steps:

[0048] Step S1: Obtain the State Grid document data and pre-process it;

[0049] Step S2: Using the TF-IDF algorithm to identify the keyword table with category information for the document name and the document title;

[0050] Step S3: According to the recognition results of the keyword table, the word2vec tool is used to select the Skip-gram model to train the word embedding model and generate keyword vectors;

[0051] Step S4: Based on the document vectorization results, an agglomerative hierarchical clustering algorithm is used to perform hierarchical clustering of the State Grid documents;

[0052] Step S5: According to the hierarchical clustering results of the State Grid documents, according to the keywords in step S2 that co-occur in the same category document set, set the corresponding category labels to complete the construction of the State Grid document hierarchical directory, so as to facilitate the State Grid users to retrieve and share the State Grid's massive documents and improve daily office efficiency.

[0053] In this embodiment, the preprocessing is as follows: word segmentation and named entity recognition of the State Grid document data are performed, and the pynlpir component of the NLPIR word segmentation system is used to perform text word segmentation in combination with the power grid entity name corpus, and part-of-speech tagging is performed at the same time, and the proprietary entity names in the State Grid documents, such as "equipment name, business name, organization name", etc., are marked, and stop word filtering is performed on the word segmentation results to delete interjections, modal particles, personal pronouns, and other words without specific document category meanings.

[0054] In this embodiment, step S2 is specifically as follows:

[0055] First, the TF-IDF algorithm is used to identify keywords with category information in the document name and document title. The formula is as follows:

[0056] TF_IDF(t)=tf(t)*idf(t);

[0057] Among them, TF is term frequency and IDF is inverse document frequency;

[0058] Then, the TF-IDF algorithm is also used to identify keywords in the document content. Finally, the first n keywords in the document name and title of the document are taken, and the m keywords in the document content are taken to form the keyword table of the document. If repeated, keyword merging is performed.

[0059] In this embodiment, step S3 obtains the document vector by averaging the word vectors of the key words:

[0060]

[0061] Among them, V(d) represents the feature vector representation of a document, and the word vector of the i-th feature word c in the text is represented by v(c1), that is, the vectorized representation of the document is the average of the sum of all keyword vectors.

[0062] In this embodiment, step S4 is specifically as follows:

[0063] Step S41: initially treating each document point as an independent cluster;

[0064] Step S42: Calculate the similarity between clusters by Euclidean distance. According to mean link and cosine similarity, find the two clusters C with the closest distance. i and C j ;

[0065] For cluster C i Contains N i sample points, whose centroid is expressed as:

[0066]

[0067] Cluster C i and C j The distance between , based on their centroids, is defined as:

[0068]

[0069] in, Cluster C j The centroid of

[0070] Step S43: Merge cluster C i and C j Become a new cluster C ij , and update the distance matrix;

[0071] Step S44: Repeat the above process until all documents are merged into one total cluster or the specified number of clusters k is reached:

[0072] Step S45: Finally, a hierarchical tree directory is formed.

[0073] In this embodiment, step S5 is specifically as follows:

[0074] After the State Grid documents are hierarchically divided by the agglomerative hierarchical clustering algorithm, multiple categories (C1, C2, …, Ck) can be obtained, and each category Ck contains several documents;

[0075] Count the keywords of all documents in category Ck and calculate the frequency of occurrence of each keyword:

[0076]

[0077] in, Indicates the number of occurrences of keyword wi in document d;

[0078] Combine TF-IDF weights to weight keyword frequency:

[0079]

[0080] Among them, TF-IDF(w i ) is the keyword wi TF-IDF value of FreqC k (w i ) is the keyword w i In Category C k The total frequency of occurrence in ;

[0081] According to Score Ck (w i ) Sort the keywords and select the top N keywords with the highest scores as category labels;

[0082] According to the hierarchical clustering results, the documents are organized into a tree-like hierarchical directory structure by category, and the extracted category labels are applied to the directory nodes. For the top-level category, the keywords of all documents in the category are selected for statistics, and high-frequency keywords are extracted as root directory labels. For each subcategory, the above keyword extraction method is repeated, and keywords are extracted based on the document set of the subcategory to generate more fine-grained labels; each document is classified into its corresponding clustering category and placed under the leaf node of the tree directory.

[0083] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0084] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0085] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1A function specified in one or more boxes.

[0086] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0087] The above is only a preferred embodiment of the present invention, and does not limit the present invention in other forms. Any technician familiar with the profession may use the above disclosed technical content to change or modify it into an equivalent embodiment with equivalent changes. However, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present invention without departing from the technical solution of the present invention still belongs to the protection scope of the technical solution of the present invention.

Claims

1. A hierarchical clustering and directory automatic construction method based on the characteristics of State Grid document data, characterized in that: The following steps are involved: Step S1: Obtain the State Grid document data and pre-process it; Step S2: Using the TF-IDF algorithm to identify the keyword table with category information for the document name and the document title; Step S3: According to the recognition results of the keyword table, the word2vec tool is used to select the Skip-gram model to train the word embedding model and generate keyword vectors; Step S4: Based on the document vectorization results, an agglomerative hierarchical clustering algorithm is used to perform hierarchical clustering of the State Grid documents; Step S5: According to the hierarchical clustering results of the State Grid documents, according to the keywords in step S2 that co-occur in the document sets of the same category, set the corresponding category labels to complete the construction of the hierarchical directory of the State Grid documents.

2. The hierarchical clustering and directory automatic construction method based on the characteristics of the State Grid document data according to claim 1 is characterized in that: The preprocessing is as follows: word segmentation and named entity recognition of the State Grid document data are performed, and the pynlpir component of the NLPIR word segmentation system is used to perform text word segmentation in combination with the power grid entity name corpus, and part-of-speech tagging is performed at the same time, marking the proprietary entity names in the State Grid documents, such as "equipment name, business name, organization name", etc., and stop word filtering is performed on the word segmentation results to delete interjections, modal particles, personal pronouns, and other words without specific document category meanings.

3. The hierarchical clustering and directory automatic construction method based on the characteristics of the State Grid document data according to claim 1 is characterized in that: The step S2 is specifically as follows: First, the TF-IDF algorithm is used to identify keywords with category information in the document name and document title. The formula is as follows: TF_IDF(t)=tf(t)*idf(t); Among them, TF is term frequency and IDF is inverse document frequency; Then, the TF-IDF algorithm is also used to identify keywords in the document content. Finally, the first n keywords in the document name and title of the document are taken, and the m keywords in the document content are taken to form the keyword table of the document. If repeated, keyword merging is performed.

4. The hierarchical clustering and directory automatic construction method based on the characteristics of the State Grid document data according to claim 1 is characterized in that: The step S3 obtains the document vector by averaging the word vectors of the key words: Among them, V(d) represents the feature vector representation of a document, and the word vector of the i-th feature word c in the text is represented by v(c1), that is, the vectorized representation of the document is the average of the sum of all keyword vectors.

5. The hierarchical clustering and directory automatic construction method based on the characteristics of the State Grid document data according to claim 1 is characterized in that: The step S4 is specifically as follows: Step S41: initially treating each document point as an independent cluster; Step S42: Calculate the similarity between clusters by Euclidean distance. According to mean link and cosine similarity, find the two clusters C with the closest distance. i and C j； For cluster C i Contains N i sample points, whose centroid is expressed as: Cluster C i and C j The distance between them, based on their centroids, is defined as: in, Cluster C j The centroid of Step S43: Merge cluster C i and C j Become a new cluster C ij , and update the distance matrix; Step S44: Repeat the above process until all documents are merged into one total cluster or the specified number of clusters k is reached: Step S45: Finally, a hierarchical tree directory is formed.

6. The hierarchical clustering and directory automatic construction method based on the characteristics of the State Grid document data according to claim 1 is characterized in that: The step S5 is specifically as follows: After the State Grid documents are hierarchically divided by the agglomerative hierarchical clustering algorithm, multiple categories (C1, C2, …, Ck) can be obtained, and each category Ck contains several documents; Count the keywords of all documents in category Ck and calculate the frequency of occurrence of each keyword: Among them, Count d (w i ) represents the number of occurrences of keyword wi in document d; Combine TF-IDF weights to weight keyword frequency: Among them, TF-IDF(w i ) is the keyword w i TF-IDF value of FreqC k (w i ) is the keyword w i In Category C k The total frequency of occurrence in ; According to Score Ck (w i ) Sort the keywords and select the top N keywords with the highest scores as category labels; According to the hierarchical clustering results, the documents are organized into a tree-like hierarchical directory structure by category, and the extracted category labels are applied to the directory nodes. For the top-level category, the keywords of all documents in the category are selected for statistics, and high-frequency keywords are extracted as root directory labels. For each subcategory, the above keyword extraction method is repeated, and keywords are extracted based on the document set of the subcategory to generate more fine-grained labels; each document is classified into its corresponding clustering category and placed under the leaf node of the tree directory.

Images