Text data driven standard system framework construction method and system

Abstract

The invention discloses a text data driven standard system framework construction method and system. The method comprises the following steps: identifying t themes from five types of texts including reports, literatures, patents, cases and standards by utilizing an LDA model, and mapping the t themes into a primary subsystem of a standard system; calculating a co-occurrence relationship of classifying the subject terms of each subject to the subject, identifying m sub-clusters according to the co-occurrence relationship by using a co-occurrence network analysis method, and mapping the m sub-clusters into a secondary subsystem of the standard system; and finally, constructing a standard system framework according to the first-level subsystem and the second-level subsystem. The input of full-chain text data ensures that a standard system framework has integrity and perspectiveness, and the combined use of an LDA topic model and a topic word co-occurrence network ensures that the standard system framework has objectivity and hierarchy.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and system for constructing a standard system framework driven by text data. Background Art

[0002] As a tool that uses systems theory to guide standardization work, the standard system can comprehensively identify existing standards and vacant standards, and provide a basis for the orderly formulation, revision and implementation of standards.

[0003] However, the current standard system framework construction method has problems such as incomplete data input and low objectivity in determining the hierarchical relationship of each subsystem due to over-reliance on "knowledge". When determining the input content of the standard system framework, most of them only analyze the existing standards, without considering more forward-looking information input such as planning and design, scientific research, technological invention and practical application, resulting in the difficulty of the constructed standard system to fully reflect the current and future standardization construction needs. When designing the standard system framework, the latest standard subject word co-occurrence network method proposed to make up for the lack of objectivity of system engineering methods such as comprehensive classification, modularization, process method, and parallel decomposition method is still difficult to determine the hierarchical relationship of the framework content elements. Therefore, it is necessary to develop a new method based on text data drive to improve the comprehensiveness and objectivity of the construction of the standard system framework. Summary of the invention

[0004] Purpose of the invention: The purpose of the present invention is to provide a simple, fast and highly operable text data-driven standard system framework construction method and system to address the deficiencies of the prior art.

[0005] Technical solution: The text data-driven standard system framework construction method of the present invention comprises the following steps:

[0006] Using LDA model, t topics are identified from five types of texts: reports, documents, patents, cases and standards;

[0007] Mapping the t topics into the first-level subsystem of the standard system;

[0008] Calculate the co-occurrence relationship between the subject words of each topic and the subject words of the topic, and identify a total of m sub-clusters of t topics using the co-occurrence network analysis method based on the co-occurrence relationship;

[0009] Mapping the m sub-clusters into secondary sub-systems of the standard system;

[0010] Construct the standard system framework based on the first-level subsystem and the second-level subsystem.

[0011] Furthermore, the construction of a standard system framework based on the primary subsystem and the secondary subsystem includes:

[0012] Similar topics in the first-level subsystems are merged, similar subclusters in the second-level subsystems are merged, and a standard system framework is constructed based on the first-level and second-level subsystems.

[0013] Furthermore, the construction of a standard system framework based on the primary subsystem and the secondary subsystem includes:

[0014] The sub-cluster contents with hierarchical relationships in the second-level subsystem are divided into the third-level subsystem, and the standard system framework is constructed based on the first-level subsystem, the second-level subsystem and the third-level subsystem.

[0015] Furthermore, the t topics identified from the five types of texts, namely, reports, documents, patents, cases and standards, using the LDA model include:

[0016] Extract the content paragraphs of the report, extract the title, abstract and keywords of the document, extract the invention name and description abstract of the patent, extract the content paragraphs of the case, extract the standard title, and establish a text database.

[0017] Furthermore, the t topics identified from the five types of texts, namely, reports, documents, patents, cases and standards, using the LDA model include:

[0018] Preprocessing the content in the text database includes:

[0019] Unify the language of the text database, establish a stop word list, a custom dictionary and a synonym replacement table, remove stop words, add custom words, and replace synonyms in the content of the text database.

[0020] Furthermore, based on the co-occurrence relationship, a total of m sub-clusters of t topics are identified using a co-occurrence network analysis method, including:

[0021] According to the co-occurrence relationship, the subject words of each topic are subjected to node filtering, and the subject words with the number of occurrences exceeding a threshold are retained.

[0022] Furthermore, based on the co-occurrence relationship, a total of m sub-clusters of t topics are identified using a co-occurrence network analysis method, including:

[0023] For each topic reported, the topic words with a word frequency greater than 4 and an edge co-occurrence number greater than 2 were retained;

[0024] For each topic of literature and patents, retain the subject terms with a frequency greater than 20 and an edge co-occurrence greater than 10;

[0025] For each topic of cases and standards, the topic terms with a frequency greater than 1 and an edge co-occurrence number greater than 1 are retained.

[0026] The text data driven standard system framework construction system of the present invention comprises:

[0027] The first-level subsystem mapping unit is used to identify t topics from five types of texts, namely, reports, documents, patents, cases and standards, using the LDA model; and map the t topics into the first-level subsystem of the standard system;

[0028] The secondary subsystem mapping unit is used to calculate the co-occurrence relationship between the subject words of each topic and the subject under the topic, and identify a total of m sub-clusters of t topics using the co-occurrence network analysis method according to the co-occurrence relationship; and map the m sub-clusters into the secondary subsystem of the standard system;

[0029] The standard system framework construction unit is used to construct the standard system framework based on the first-level subsystem and the second-level subsystem.

[0030] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and is characterized in that when the computer program is loaded into the processor, the text data-driven standard system framework construction method is implemented.

[0031] The computer-readable storage medium of the present invention stores a computer program, and is characterized in that when the computer program is executed by a processor, the text data-driven standard system framework construction method is implemented.

[0032] Beneficial effect: Compared with the prior art, the advantages of the present invention are: the present invention discloses a multi-source heterogeneous text layer-by-layer mining method based on the input of the full-chain text data of "planning and design-scientific research-technological invention-practical application-standard formulation", integrating the LDA topic model and the subject word co-occurrence network. The input of the full-chain text data can ensure that the standard system framework is complete and forward-looking, the LDA topic modeling method can effectively cluster a large number of documents, and the subject word co-occurrence network method can further mine sub-clusters of documents with the same subject on this basis. The combined use of the two methods is expected to provide method support for the scientific and objective determination of the hierarchical relationship of subsystems at all levels of the standard system, which can ensure that the standard system framework is objective and hierarchical. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 A flow chart of the method for constructing the standard system framework of the present invention;

[0034] Figure 2 Schematic diagram of the distribution of five types of text data topics in an embodiment of the present invention;

[0035] Figure 3 This is a schematic diagram of the identification results of each theme sub-cluster of the planning and design layer in an embodiment of the present invention;

[0036] Figure 4 This is a schematic diagram of the identification results of each subject sub-cluster at the scientific research layer in an embodiment of the present invention;

[0037] Figure 5 A schematic diagram of the identification results of each subject sub-cluster at the technical invention layer in an embodiment of the present invention;

[0038] Figure 6 This is a schematic diagram of the identification results of each subject sub-cluster at the actual application layer in an embodiment of the present invention;

[0039] Figure 7 This is a schematic diagram of the identification results of each subject sub-cluster of the standard setting layer in an embodiment of the present invention;

[0040] Figure 8 A schematic diagram of a low-carbon operation standard system framework for a municipal sewage treatment plant constructed using the text data-driven standard system framework construction method of the present invention in an embodiment of the present invention;

[0041] Fig. 9 This is a schematic diagram of the framework of a low-carbon operation standard system for urban sewage treatment plants built using existing common methods in an embodiment of the present invention. DETAILED DESCRIPTION

[0042] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0043] like Figure 1 As shown, the text data-driven standard system framework construction method includes the following steps.

[0044] Step 1: Determine the search ideas and time limits for reports, documents, patents, cases and standards, and build five types of text databases for reports, documents, patents, cases and standards based on the text search ideas and time limits. Reports, documents, patents and cases can represent specific field planning and design, scientific research, technological inventions and practical applications, and their introduction can provide complete data input for comprehensively reflecting the current status and future development trends of standards.

[0045] In the text retrieval approach, report texts are retrieved and screened on official government and industry websites to extract content-related paragraphs; document texts are retrieved and screened on core journals after determining the search keywords to extract English titles, abstracts, and keywords of document texts; patent texts are retrieved and screened on patent databases after determining the search keywords to extract English titles and abstracts of patent texts; case texts are obtained by further mining cases in report texts and document texts to extract paragraphs describing the cases; standard texts are retrieved and screened on official websites of standard information platforms at all levels after determining the search keywords to extract English titles of standard texts.

[0046] Step 2: pre-process the text data in the five types of text databases by unifying the language, segmenting the words, restoring the parts of speech, removing stop words, etc.

[0047] In the preprocessing of text data, English is used as the unified language of each text database; a general initial stop word list is downloaded, and meaningless words are continuously identified based on the word segmentation results and added to the stop word list; all keywords in the literature text database are summarized to form a custom dictionary for word segmentation, and fixed phrases are supplemented by manually reading the text to filter them; a synonym replacement table is constructed to perform synonym replacement on the word segmentation results.

[0048] Step 3: Perform LDA topic modeling on the five types of preprocessed text data respectively, identify a total of t topics of the five types of text data, and map the t topics into the first-level subsystem of the standard system.

[0049] When LDA topic modeling is performed, the two indicators of perplexity and consistency are used comprehensively to determine the optimal number of topics, and professional terms are used manually to label each topic based on the 10 most frequent keywords.

[0050] Step 4: Based on the t topics identified, use the subject word co-occurrence network analysis method to further conduct in-depth mining, identify a total of m sub-clusters of the t topics, and map the m sub-cluster extraction results into the secondary subsystem of the standard system.

[0051] When analyzing the co-occurrence network of subject words, the co-occurrence relationships in each topic of each text were filtered according to the proportion of the number of subject words in the five categories of text, so as to ensure that the sparsity of each network is consistent as much as possible; after node filtering, Gephi modular detection technology is used to identify network sub-clusters.

[0052] Step 5: After manual interpretation, similar topics or similar sub-clusters of each layer of subsystem are merged to construct a standard system framework.

[0053] This embodiment takes the construction of a low-carbon operation standard system framework for a municipal sewage treatment plant as an example to further illustrate the method of the present invention.

[0054] 1: Construct a database of reports, documents, patents, cases and standard texts on low-carbon operation of urban sewage treatment plants as the content input of the framework of the low-carbon operation standard system for urban sewage treatment plants. The search period is from January 2008 to December 2022, including:

[0055] 1-1: Search and screen 50 relevant report texts on official government and industry websites, and extract relevant paragraphs.

[0056] 1-2: Search and screen 398 articles in CNKI core journals and Web of Science core journals, and extract the English titles, abstracts, and keywords of the article texts.

[0057] 1-3: Search and screen 413 patents in CNKI patent database, Derwent patent database and Espacenet European Patent Office respectively, and extract the English invention title and abstract of the patent text.

[0058] 1-4: Further mining the report text and literature text, obtaining 160 case texts, and extracting paragraphs describing the cases.

[0059] 1-5: Search and screen 142 standards on the National Standard Information Public Service Platform, ANSI, CEN and ISO official websites, and extract the English titles of the standard texts.

[0060] Step 2: Based on step 1, use the NLTK library to perform data preprocessing on the five types of text databases, including:

[0061] 2-1: Use online translation software to make English the unified language for each text database.

[0062] 2-2: Download the initial stop word list from the CSDN website, and then continuously identify meaningless words based on the word segmentation results and add them to the stop word list. The final stop word list contains a total of 1,690 stop words.

[0063] 2-3: All keywords in the literature text database are aggregated to form a custom dictionary for word segmentation. At the same time, fixed phrases are selected through manual reading of the text to supplement the custom dictionary, which contains a total of 4436 phrases.

[0064] 2-4: Construct a synonym replacement table to perform synonym replacement on the segmentation results. For example, replace "solarpv panels", "solar photovoltaic", "solar light" and "solar generation" with "solarpower". The synonym replacement table contains 371 pairs of replacement words or replacement phrases.

[0065] Step 3: Based on step 2, LDA topic modeling is performed on the five types of preprocessed text data, specifically including:

[0066] 3-1: The number of topics traversed by the LDA model is set to 2 to 10. Based on the two indicators of topic perplexity and topic consistency, 3, 3, 5, 4, and 4 topics of the five types of text data are identified respectively, totaling 19 topics.

[0067] 3-2: Based on the 10 most frequent keywords in each subject, manually use professional terms to label each topic, such as Figure 2 As shown in (a) to (e) in the figure. Figure 2(a) to (e) are the themes of reports, documents, patents, cases and standards on low-carbon operation of urban sewage treatment plants. The five types of texts, reports, documents, patents, cases and standards, correspond to the planning and design layer, scientific research layer, technological invention layer, practical application layer and standard formulation layer of the standard system framework.

[0068] 3-3: Map the 19 topics in total into the first-level subsystem of the low-carbon operation standard system framework for urban sewage treatment plants.

[0069] Step 4: Based on step 3, the subject word co-occurrence network analysis method is used to further explore the 19 topics, including:

[0070] 4-1: Calculate the co-occurrence relationship of the subject words in the 19 topics in the text data classified under the topic.

[0071] 4-2: Based on the co-occurrence relationship, the co-occurrence relationship in each topic of each text is filtered according to the ratio of the number of subject words in the five categories of text (approximately 2:10:10:1:1). For each topic of the report text, the information with a word frequency greater than 4 and an edge co-occurrence number greater than 2 is retained; for each topic of the literature and patent text, the information with a word frequency greater than 20 and an edge co-occurrence number greater than 10 is retained; for each topic of the case and standard text, the information with a word frequency greater than 1 and an edge co-occurrence number greater than 1 is retained.

[0072] 4-3: Based on the filtered edge point information, a co-occurrence network of keywords for each topic is constructed, and the modular detection technology of Gephi0.10.1 is used to identify network subclusters, and 3, 3, 3, 4, 4, 3, 3, 2, 3, 3, 3, 3, 2, 2, 3, 2, 3, 2, 3, 2, 3 subclusters are identified respectively, for a total of 54 subclusters. The subcluster identification results of 19 topics in the planning and design layer, scientific research layer, technological invention layer, practical application layer, and standard setting layer are as follows: Figures 3 to 7 shown. Figure 3 (a) is the sub-clustering result of the RT1 energy substitution theme. Figure 3 (b) is the sub-clustering result of the RT2 sludge energy and resource recovery theme. Figure 3 Middle (c) is the sub-clustering result of the RT3 carbon footprint theme; Figure 4 (a) is the sub-clustering result of the JT1 energy-saving and consumption-reducing technology theme. Figure 4 (b) is the sub-clustering result of the JT2 carbon emission management theme. Figure 4 Middle (c) is the sub-clustering result of the JT3 low-carbon treatment technology theme; Figure 5 (a) is the sub-clustering result of the PT1 pipeline equipment theme. Figure 5 (b) is the sub-clustering result of the PT2 energy and resource recovery equipment theme. Figure 5(c) is the sub-clustering result of the PT3 biological nitrogen removal and phosphorus removal theme. Figure 5 (d) is the sub-clustering result of the PT4 new energy development equipment theme. Figure 5 Middle (e) is the sub-clustering result of the PT5 low-carbon treatment equipment theme; Figure 6 (a) is the sub-clustering result of the CT1 chemical energy recovery theme. Figure 6 (b) is the sub-clustering result of the CT2 thermal energy recovery topic. Figure 6 (c) is the sub-clustering result of the CT3 cogeneration theme. Figure 6 Middle (d) is the sub-clustering result of CT4 resource recycling theme; Figure 7 (a) is the sub-clustering result of the ST1 sludge comprehensive utilization theme. Figure 7 (b) is the sub-clustering result of the ST2 energy saving and energy efficiency improvement theme. Figure 7 (c) is the sub-clustering result of the ST3 sewage comprehensive utilization theme. Figure 7 (d) is the sub-clustering result of the ST4 smart operation and maintenance theme.

[0073] 4-4: Map the 54 sub-clusters as the secondary sub-system of the low-carbon operation standard system framework of the urban sewage treatment plant.

[0074] Step 5: Based on Steps 3 and 4, similar topics in the first-level subsystems are merged, and similar sub-clusters in the second-level subsystems are merged. Sub-clusters in the second-level subsystems that have hierarchical relationships in content are divided into third-level subsystems after manual interpretation. Finally, a low-carbon operation standard system framework for urban sewage treatment plants was established, including 7 first-level subsystems, 28 second-level subsystems, and 18 third-level subsystems, such as Figure 8 shown.

[0075] If the existing common method is used to construct the standard system framework for low-carbon operation of urban sewage treatment plants, that is, only the standard text is input as the content of the standard system framework, and only the subject word co-occurrence network analysis method is used for primary mining, then only the low-carbon operation standard system framework for urban sewage treatment plants with 9 primary subsystems as the content can be obtained, such as Fig. 9 In contrast, the standard system framework built using this method includes 7 first-level subsystems, 28 second-level subsystems, and 18 third-level subsystems, which obviously has the advantages of more comprehensive content and clearer levels.

[0076] The text data driven standard system framework construction system of the present invention comprises:

[0077] The first-level subsystem mapping unit is used to identify t topics from five types of texts, namely, reports, documents, patents, cases and standards, using the LDA model; and map the t topics into the first-level subsystem of the standard system;

[0078] A secondary subsystem mapping unit is used to identify m subclusters of t topics based on the co-occurrence relationship of the subject words of each topic in the text data classified under the topic by using a co-occurrence network analysis method; and map the m subclusters into a secondary subsystem of the standard system;

[0079] The standard system framework construction unit is used to construct the standard system framework based on the first-level subsystem and the second-level subsystem.

[0080] The electronic device described in the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and is characterized in that when the computer program is loaded into the processor, the text data-driven standard system framework construction method is implemented.

[0081] The computer-readable storage medium of the present invention stores a computer program, and is characterized in that when the computer program is executed by a processor, the text data-driven standard system framework construction method is implemented.

[0082] The computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other medium that can be used to store program code in the form of instructions or data structures and that can be accessed by a computer.

[0083] The processor is used to execute the computer program stored in the memory to implement each step of the method involved in the above embodiment.

Claims

1. A method for constructing a standard system framework driven by text data, characterized in that: The steps include: Using LDA model, t topics are identified from five types of texts: reports, documents, patents, cases and standards; Mapping the t topics into the first-level subsystems of the standard system; Calculate the co-occurrence relationship between the subject words of each topic and the subject words of the topic, and identify a total of m sub-clusters of t topics using the co-occurrence network analysis method based on the co-occurrence relationship; Mapping the m sub-clusters into secondary sub-systems of the standard system; Construct the standard system framework based on the first-level subsystem and the second-level subsystem.

2. The method for constructing a text data-driven standard system framework according to claim 1, characterized in that: The standard system framework constructed according to the first-level subsystem and the second-level subsystem includes: Similar topics in the first-level subsystems are merged, similar subclusters in the second-level subsystems are merged, and a standard system framework is constructed based on the first-level and second-level subsystems.

3. The method for constructing a text data-driven standard system framework according to claim 1, characterized in that: The standard system framework constructed according to the first-level subsystem and the second-level subsystem includes: The sub-cluster contents with hierarchical relationships in the second-level subsystem are divided into the third-level subsystem, and the standard system framework is constructed based on the first-level subsystem, the second-level subsystem and the third-level subsystem.

4. The method for constructing a text data-driven standard system framework according to claim 1, characterized in that: The topics identified by using the LDA model from the five types of texts, namely reports, documents, patents, cases and standards, include: Extract the content paragraphs of the report, extract the title, abstract and keywords of the document, extract the invention name and description abstract of the patent, extract the content paragraphs of the case, extract the standard title, and establish a text database.

5. The method for constructing a text data-driven standard system framework according to claim 4, characterized in that: The topics identified by using the LDA model from the five types of texts, namely reports, documents, patents, cases and standards, include: Preprocessing the content in the text database includes: Unify the language of the text database, establish a stop word list, a custom dictionary and a synonym replacement table, remove stop words, add custom words, and replace synonyms in the content of the text database.

6. The method for constructing a text data-driven standard system framework according to claim 1, characterized in that: Based on the co-occurrence relationship, a total of m sub-clusters of t topics are identified using co-occurrence network analysis, including: According to the co-occurrence relationship, the subject words of each topic are subjected to node filtering, and the subject words with the number of occurrences exceeding a threshold are retained.

7. The method for constructing a text data-driven standard system framework according to claim 6, characterized in that: Based on the co-occurrence relationship, a total of m sub-clusters of t topics are identified using co-occurrence network analysis, including: For each topic reported, the topic words with a word frequency greater than 4 and an edge co-occurrence number greater than 2 were retained; For each topic of literature and patents, retain the subject terms with a frequency greater than 20 and an edge co-occurrence greater than 10; For each topic of cases and standards, the topic terms with a frequency greater than 1 and an edge co-occurrence number greater than 1 are retained.

8. A text data driven standard system framework construction system, characterized in that: include: The first-level subsystem mapping unit is used to identify t topics from five types of texts, namely, reports, documents, patents, cases and standards, using the LDA model; and map the t topics into the first-level subsystem of the standard system; The secondary subsystem mapping unit is used to calculate the co-occurrence relationship between the subject words of each topic and the subject under the topic, and identify a total of m sub-clusters of t topics using the co-occurrence network analysis method according to the co-occurrence relationship; and map the m sub-clusters into the secondary subsystem of the standard system; The standard system framework construction unit is used to construct the standard system framework based on the first-level subsystem and the second-level subsystem.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is loaded into a processor, the method for constructing a text data-driven standard system framework according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for constructing a text data-driven standard system framework according to any one of claims 1 to 7 is implemented.

Images