Water conservancy construction hidden danger theme mining method based on LDA model

Abstract

The invention discloses a water conservancy construction hidden danger theme mining method based on an LDA model, and the method comprises the steps: obtaining water conservancy construction hidden danger investigation record text data, building a sample library, carrying out the word segmentation of the text data to build a customized dictionary and a deactivation dictionary, calculating the weight of feature words through a TF-IDF algorithm, screening high-frequency feature words, and carrying out the recognition of the high-frequency feature words. The method comprises the steps of determining the optimal topic number of an LDA model by combining the confusion degree and topic consistency, training the LDA model by means of a Gibbs sampling method to obtain a topic-word distribution matrix, carrying out manual classification and supplement on topics to form a standardized hidden danger topic system, and carrying out validity verification on a mining result by utilizing a verification sample. According to the method, the high-frequency hidden danger theme can be automatically mined from the unstructured hidden danger text, the problems of low manual classification efficiency and high subjectivity are effectively solved, the accuracy and systematicness of hidden danger investigation and treatment are improved, and powerful data support is provided for water conservancy construction safety management.

Description

Technical Field

[0001] This invention relates to the field of water conservancy project safety production management technology, and is particularly applicable to the method of identifying potential hazards in water conservancy construction based on the LDA model. Background Technology

[0002] In water conservancy engineering construction, the effective identification and classification of construction hazards is the foundation of safety production management. Traditional manual classification relies on experience and suffers from numerous technical problems. For example, it is difficult to process unstructured text; the inconsistent expressions and formats of records filled out by managers lead to low efficiency and subjectivity in manual classification, affecting the efficiency of hazard management; the lack of unified standards for hazard classification leads to inconsistent classification results for different projects and personnel, making it difficult to accumulate systematic safety management data and hindering continuous management improvement; manual methods struggle to quickly extract high-frequency hazard themes from massive amounts of data, resulting in unreasonable allocation of safety management resources, delayed identification of high-frequency hazards, and increased risks; and the weak correlation with accident causes, without incorporating accident causation theories, makes it difficult to trace the root cause of accidents and form closed-loop management. Existing machine learning methods also have shortcomings in the application of construction hazard classification, such as the inability to optimize parameters for the characteristics of the water conservancy construction industry, and are difficult to directly apply to on-site management. Therefore, there is an urgent need for an automatic hazard theme mining, classification, and verification method adapted to the characteristics of the water conservancy construction industry to improve the accuracy and efficiency of safety management. Summary of the Invention

[0003] The purpose of this invention is to provide a method for mining potential hazards in water conservancy construction based on the LDA model, which can solve the problem that potential hazards in water conservancy projects cannot be automatically mined and classified.

[0004] To achieve the above objectives, the method for identifying potential hazards in water conservancy construction based on the LDA model described in this invention includes the following steps: Data acquisition and sample library construction; collecting text records of hidden danger investigation in water conservancy projects, and dividing them into training samples and validation samples; Text preprocessing; Configure a custom dictionary and a deactivated dictionary for construction hazards, and use the Jieba library to perform word segmentation on the hazard investigation record text based on the custom dictionary and the deactivated dictionary to obtain the word segmentation of the hazard investigation record text; The selection of construction hazard topics: The TF-IDF weight value of each word after word segmentation is calculated using the TF-IDF algorithm. The TF-IDF weight values ​​are divided into high-score segments and low-score segments. Special words in the low-score segments of TF-IDF weight values ​​are deleted, and words in the high-score segments of TF-IDF values ​​are retained as construction hazard topics. Determine the optimal number of topics; select different numbers of topics, construct LDA models for each number of topics, calculate the perplexity and topic consistency score for each model, plot the combined curve of topic consistency and perplexity, and select the number of topics with perplexity lower than the preset value and topic consistency higher than the preset value as the optimal number of topics; LDA Model Construction and Training: Input the vocabulary with high TF-IDF weight values ​​into the LDA model to train the LDA model. Based on the determined optimal number of topics, use the Gibbs sampling method to estimate the latent variables in the LDA model and solve for the distribution matrix of potential topics and vocabulary in the LDA model. Topic mining and classification: Based on the lexical descriptions in the lexical distribution matrix, topic names are summarized, noisy topics are manually deleted and missing topics are added, and content-related topics are merged to form a standardized hazard topic system.

[0005] Furthermore, the training samples are derived from the hazard investigation records of water conservancy engineering general contracting projects; the verification samples are taken from the hazard investigation records of water conservancy engineering supervision projects; the samples cover water diversion and regulation, reservoir hub projects, river management, irrigation and drainage, and reinforcement projects.

[0006] Furthermore, the effectiveness of the theme system was confirmed by comparing the distribution rate and proportion of potential hazards in the training and validation samples.

[0007] Furthermore, it also includes analyzing the distribution characteristics of each hidden danger theme; analyzing the distribution characteristics of each hidden danger theme from the perspectives of individual behavior, equipment and facilities, and organizational management, and clarifying the focus of each hidden danger theme in safety production management; differential analysis and generation of preventive measures; and proposing accident prevention measures from the perspectives of rapid handling of visible hidden dangers and prevention and control of potential risks by comparing the distribution differences of accident causative factors and each hidden danger theme.

[0008] Furthermore, the formula for calculating the TF-IDF weight value of each word after word segmentation is as follows: In the formula: Vocabulary after word segmentation for hazard investigation record text The weight value; For vocabulary Text dataset of hazard investigation records The number of times in; Text dataset for hazard investigation records The number of times each word appears; For the k-th word; The total number of documents in the text dataset recording hazard investigation records; For words The number of documents.

[0009] Furthermore, the optimal number of topics is set to the number of topics sampled by Gibbs, with hyperparameters α and β set to auto. The iteration is repeated several times until Gibbs sampling converges. The conditional probability of a word belonging to each potential topic is calculated by updating the potential topic assignment for each word.

[0010] Furthermore, the distribution characteristics of each potential hazard theme are analyzed, specifically the distribution proportions of individual behavior-related themes, equipment and facility-related themes, and organizational management-related themes, to provide a basis for the allocation of safety management resources.

[0011] Furthermore, the differential analysis and prevention measures generation includes comparing the proportion of organizational and management factors in accident causes with the distribution rate of organizational and management themes in hidden danger themes, analyzing the management gap between explicit hidden dangers and potential risks, and proposing optimization measures for the dual prevention mechanism, including constructing a database linking hazard sources and construction sites and a high-frequency hidden danger knowledge base.

[0012] The advantages of this invention lie in its ability to construct a sample library by collecting hazard investigation records, segmenting words using a custom dictionary, filtering feature words using the TF-IDF algorithm, determining the optimal number of topics for the LDA model by combining perplexity and topic consistency, training the model using Gibbs sampling to obtain a topic-word distribution matrix, manually optimizing it to form a standardized topic system, and finally verifying and analyzing topic features to generate preventive measures. This invention solves the problems of low efficiency and strong subjectivity in manual classification, and can automatically mine high-frequency hazard topics from unstructured text, improving the accuracy and systematicness of hazard investigation and management, and providing data support for water conservancy construction safety management. Attached Figure Description

[0013] Figure 1 This is a flowchart of a method for identifying potential hazards in water conservancy construction based on an LDA model, according to the present invention.

[0014] Figure 2 This is a combined curve of thematic consistency and confusion of the present invention. Detailed Implementation

[0015] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0016] like Figure 1 As shown, the method for identifying potential hazards in water conservancy construction based on the LDA model described in this invention includes the following steps: Data Acquisition and Sample Library Construction: Textual records of hazard investigations in water conservancy project construction were collected and divided into training and validation samples. Training samples were derived from hazard investigation records of general contracting projects in water conservancy engineering, while validation samples were derived from hazard investigation records of supervision projects in water conservancy engineering. The project types of the samples covered various engineering types, including water diversion, reservoir projects, river management, irrigation and drainage, and reinforcement. This step provided fundamental data support for subsequent text processing and model training, and the diverse sources and types of samples ensured the diversity and representativeness of the data.

[0017] In this embodiment, two sample libraries are established: Sample 1 serves as the model training sample, and Sample 2 serves as the validation sample, used to verify the representativeness of the hazard themes and distribution characteristics identified by Sample 1. Sample 1 originates from 3,410 hazard investigation records collected from 10 water conservancy engineering general contracting projects between 2023 and 2024. Sample 2 relies on 4,765 hazard investigation records collected from 12 construction units across 8 water conservancy engineering supervision projects between 2023 and 2024. The project types of the sample sources cover water diversion, reservoir hub projects, river management, irrigation and drainage, and reinforcement projects.

[0018] Text preprocessing: Configure a custom dictionary and a stop dictionary for construction hazards. Based on the custom dictionary and the stop dictionary, use the Jieba library to perform word segmentation on the hazard investigation record text to obtain the word segmentation results. This step relies on the data in the previously built sample library to perform preliminary text processing, remove meaningless words, and make subsequent feature word extraction more accurate.

[0019] In this embodiment, to ensure the accuracy of Jieba's word segmentation results, it is necessary to configure a custom dictionary for construction hazards, a stop dictionary, and select an appropriate word segmentation method. The custom dictionary for construction hazards is derived from six parts: "Special Terms for Safe Production," "Engineering Supervision," "Common Terms for Water Conservancy Engineering," "Complete Glossary of Civil Engineering Terms," ​​"Comprehensive Engineering Thesaurus," and custom terms from the Sogou Dictionary, forming a custom dictionary of 26,030 words, effectively preventing terms from being incorrectly separated or merged. To further remove noisy words, a stop dictionary is used, including the Baidu stop word dictionary, the Harbin Institute of Technology stop word dictionary, the Sichuan University Machine Intelligence Laboratory stop word dictionary, the Fudan University stop word dictionary, and meaningless words such as place names at the county level and above nationwide, adverbs, conjunctions, and interjections, forming a stop word dictionary of 3,915 words. Word segmentation is implemented using the Python programming language, and some results are shown in Table 1, generating a total of 16,257 words.

[0020] Table 1. Word segmentation results of the hidden danger investigation record (partial) Selection of Construction Hazard Topics: The TF-IDF weight value of each segmented word is calculated using the TF-IDF algorithm. The TF-IDF weight values ​​are then divided into high-scoring and low-scoring segments. Words with low TF-IDF weight values ​​are deleted, and words with high TF-IDF weight values ​​are retained as the construction hazard topics. This step, based on the preprocessed word segmentation results, uses the algorithm to select words (i.e., feature words) that can represent the hazard topics, preparing for subsequent model training.

[0021] In this embodiment, the TF-IDF function is currently the most widely used vector space model, which considers word frequency changes while also distinguishing text semantic features to a certain extent. In TF-IDF, TF calculates the frequency of a word in the incident dataset, and IDF calculates the ratio of the number of documents in the incident dataset to the number of times a specific word appears in the incident dataset. The calculation formula is as follows: In the formula: Vocabulary after word segmentation of the hidden danger text The weights; For vocabulary Text dataset of hazard investigation records The number of times in; Text dataset for hazard investigation records The number of times each word appears; For the k-th word; The total number of documents in the text dataset recording hazard investigation records; For words The number of documents.

[0022] After segmenting the hazard investigation record text using Jieba, 16,257 words were obtained. Many of these words were meaningless in revealing the hidden dangers in water conservancy project construction and needed to be further removed. Therefore, this algorithm was used to screen feature words. Based on Python, the TF-IDF algorithm was used to calculate the word weight value, and words with low TF-IDF weight values ​​were deleted. The TF-IDF values ​​of some words are shown in Tables 2 and 3.

[0023] Table 2. Vocabulary with high TF-IDF weights (partial list) Table 3. Vocabulary with low TF-IDF weights (partial list) By comparing the TF-IDF weight values ​​of words, it can be found that words with higher TF-IDF weight values ​​reveal the text topic better than words with lower TF-IDF weight values. Therefore, in order to improve the text clustering effect of the LDA topic model, words with lower TF-IDF weight values ​​are deleted from the hazard investigation record text, and the remaining feature words are input into the LDA topic model for training.

[0024] Determining the optimal number of topics: Different numbers of topics are selected, and LDA models are built for each number of topics. The perplexity and topic consistency scores of each model are calculated, and a combined curve of topic consistency and perplexity is plotted. The optimal number of topics is determined by selecting the number of topics with perplexity below a preset value and topic consistency above a preset value. This step is to determine the optimal parameters of the LDA model, enabling the model to better uncover potential topics. The results will directly affect the subsequent construction and training of the LDA model.

[0025] In this embodiment, topic consistency and perplexity curves are plotted on the same graph, with the horizontal axis representing the number of topics and the vertical axes representing topic consistency and perplexity, respectively. By observing the combined curve, the number of topics that results in low perplexity and high topic consistency is selected. Before constructing and solving the LDA topic model, the optimal number of topics is determined by combining perplexity and topic consistency, with the topic number K values ​​being 5, 10, 15, 20, 25, 30, 35, 45... The α and β parameters in the Gensim library's LDA model are set to auto, which learns asymmetric priors based on actual data. This setting performs well in many experiments when the number of topics is large. The number of sampling iterations is set to 1000 to obtain topic consistency and perplexity under different LDA topic models. Figure 2 This is a combined curve of topic consistency and confusion under different LDA topic models. From... Figure 2 As can be seen, the confusion level tends to plateau when the number of topics increases from 20 onwards. While the confusion level decreases as the number of topics increases, too many topic categories clearly contradict the original intention of hazard classification. Looking at the consistency score curve, the consistency score reaches a peak when the number of topics is 20. Combining these two factors, the estimated optimal number of topics is 20.

[0026] LDA Model Construction and Training: Input the feature words with high TF-IDF weight values ​​into the LDA model to train the LDA model. Based on the determined optimal number of topics, use the Gibbs sampling method to estimate the latent variables in the LDA model and solve for the distribution matrix of hidden topic-vocabulary (feature words) of the LDA model.

[0027] In the Gibbs sampling process, the optimal number of topics is set to the number of topics sampled by Gibbs, i.e., 20, and the hyperparameters α and β are set to auto. The process is iterated several times until Gibbs sampling converges. By updating the topic assignment of each word, the conditional probability of a word belonging to each topic is calculated, as shown in the following formula: in, This represents all topic assignments except for the nth word in document d. Indicates all words, This indicates that all words represent the theme. Hyperparameters representing the topic distribution of words. Hyperparameters representing the topic distribution of words represent topics. Chinese words The number of times it appears, Indicates the size of the vocabulary. Indicates the topic The number of times all words appear in the text. Hyperparameters representing the topic distribution of documents Document Chinese theme The number of times it appears, Indicates the total number of documents. Show topics in all documents The number of times it appears.

[0028] As shown in Table 4, the results of the hazard theme mining are obtained. In this embodiment, each theme contains 7 words.

[0029] Table 4. Theme Mining Results Topic Mining and Classification: Based on the feature words in the topic-vocabulary distribution matrix, topic names are summarized. Noisy topics are manually removed and missing topics are added. Content-related topics are merged to form a standardized potential hazard topic system. This step relies on the previously determined optimal number of topics and the selected feature words. Through training, a matrix that can describe the relationship between potential hazard topics and vocabulary is obtained.

[0030] In this embodiment, 20 hazard themes were calculated using the LDA topic model. Further manual screening removed four noise themes. Theme names were defined based on the main keywords under each theme. Keywords under each theme were deleted and added, and themes related to the content were merged. For example, those related to hot work and hazardous materials management were merged into fire safety, resulting in 13 themes, as shown in Table 5.

[0031] Table 5. Theme Mining Results Using the Pandas library, fields were extracted from the segmented hazard descriptions. The logic was set so that each hazard could only match one topic; when multiple topics could be matched, the topic with the most matching keywords was used. The distribution characteristics of these 13 topics in the hazard log text (Sample 1) are shown in Table 6.

[0032] Table 6. Distribution characteristics of 13 themes in the hazard log text (sample 1) As can be seen from Table 6, these 13 themes can cover nearly 90% of the potential hazards, indicating that the LDA theme model can effectively and quickly and accurately extract potential hazard themes from massive amounts of data.

[0033] Results Validation: The representativeness of the hazard theme distribution characteristics obtained from the mining was verified using validation samples. By comparing the theme distribution rate order and proportion between the training samples and the validation samples, the effectiveness of the theme system was confirmed. This step is to verify whether the previously constructed theme system has universality and reliability, and it depends on the previously constructed sample library and the mined theme system.

[0034] To further verify the representativeness of the hazard themes and distributions mined from Sample 1, Sample 2 was segmented into words. Using the Pandas library, fields were extracted from the segmented hazard descriptions, and the hazard theme distribution is shown in Table 7.

[0035] Table 7. Distribution characteristics of potential hazards in Sample 2 As can be seen from Table 7, considering noise and individual potential hazards, the 20 themes identified in Sample 1 can also effectively reveal the potential hazard themes in Sample 2, indicating that these 20 themes are highly representative.

[0036] Comparing Tables 6 and 7, it can be found that the distribution order of various themes is roughly the same in the two samples, with slight differences. This is due to the different construction operations of different projects.

[0037] In other embodiments, the method also includes analyzing the distribution characteristics of each potential hazard theme. Specifically, this involves analyzing the distribution characteristics of each theme from the perspectives of individual behavior, equipment and facilities, and organizational management. This includes statistically analyzing the distribution proportions of individual behavior-related themes, equipment and facilities-related themes, and organizational management-related themes, identifying high-frequency potential hazard themes and their emphasis in safety production management. By analyzing the distribution characteristics of potential hazard themes, direction can be provided for safety management, safety management resources can be allocated rationally, and management efficiency can be improved.

[0038] In other embodiments, the process also includes differentiated analysis and prevention measure generation: by comparing the distribution differences of accident causative factors and various hazard themes, accident prevention measures are proposed from the perspectives of rapid handling of visible hazards and control of potential risks. Specifically, by comparing the proportion of organizational and management factors in accident causation with the distribution rate of organizational and management themes in hazard themes, the management gap between visible hazards and potential risks is analyzed, and optimization measures for the dual prevention mechanism are proposed, including constructing a database linking hazard sources and construction sites and a high-frequency hazard knowledge base. This step, based on hazard theme analysis and combined with accident causative factors, proposes targeted prevention measures to achieve closed-loop management from hazard identification to accident prevention, helping to identify weaknesses in management, optimize prevention mechanisms, and improve safety management levels.

[0039] The above processing steps reveal that the most common potential hazards include temporary electricity use, fire safety, edge protection, civilized construction, signage, safety passages, scaffolding erection, hoisting and lifting, personal protective equipment, and earthwork operations. These ten categories account for over 80% of all hazards, with each category having a probability of occurrence greater than 2.5%. From the perspectives of individual behavior, equipment and facilities, and organizational management, these three aspects account for (6.13% and 3.84%), (82.26% and 85.49%), and (1.95% and 2.39%) of hazards in Sample 1 and Sample 2, respectively. This indicates that hazard identification and rectification practices focus primarily on unsafe human behavior and unsafe conditions of equipment, with less attention paid to organizational management and other factors.

[0040] This invention proposes a method for mining potential hazards in water conservancy construction based on the LDA model. A sample library is constructed by collecting hazard investigation records. Feature words are selected through custom dictionary segmentation and TF-IDF algorithm. The optimal number of topics for the LDA model is determined by combining perplexity and topic consistency. The model is then trained using Gibbs sampling to obtain a topic-word distribution matrix, which is manually optimized to form a standardized topic system. Finally, the topic features are verified and analyzed to generate preventative measures. This method utilizes the LDA topic model and TF-IDF algorithm to solve the problems of low efficiency and strong subjectivity in manual classification. It can automatically mine high-frequency hazard topics from unstructured text, improving the accuracy and systematic nature of hazard investigation and management, and providing data support for water conservancy construction safety management.

[0041] Those skilled in the art should understand that the above embodiments are merely illustrative of the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for identifying potential hazards in hydraulic construction based on an LDA model, characterized in that, Includes the following steps: Data acquisition and sample library construction; collecting text records of hidden danger investigation in water conservancy projects, and dividing them into training samples and validation samples; Text preprocessing; Configure a custom dictionary and a deactivated dictionary for construction hazards, and use the Jieba library to perform word segmentation on the hazard investigation record text based on the custom dictionary and the deactivated dictionary to obtain the word segmentation of the hazard investigation record text; The selection of construction hazard topics: The TF-IDF weight value of each word after word segmentation is calculated using the TF-IDF algorithm. The TF-IDF weight values ​​are divided into high-score segments and low-score segments. Special words in the low-score segments of TF-IDF weight values ​​are deleted, and words in the high-score segments of TF-IDF values ​​are retained as construction hazard topics. Determine the optimal number of topics; select different numbers of topics, construct LDA models for each number of topics, calculate the perplexity and topic consistency score for each model, plot the combined curve of topic consistency and perplexity, and select the number of topics with perplexity lower than the preset value and topic consistency higher than the preset value as the optimal number of topics; LDA Model Construction and Training: Input the vocabulary with high TF-IDF weight values ​​into the LDA model to train the LDA model. Based on the determined optimal number of topics, use the Gibbs sampling method to estimate the latent variables in the LDA model and solve for the distribution matrix of potential topics and vocabulary in the LDA model. Topic mining and classification: Based on the lexical descriptions in the lexical distribution matrix, topic names are summarized, noisy topics are manually deleted and missing topics are added, and content-related topics are merged to form a standardized hazard topic system.

2. The method for identifying potential hazards in water conservancy construction based on the LDA model according to claim 1, characterized in that: The training samples are derived from the hidden danger investigation records of water conservancy engineering general contracting projects; the verification samples are taken from the hidden danger investigation records of water conservancy engineering supervision projects; the samples cover water diversion and diversion, reservoir hub projects, river management, irrigation and drainage, and reinforcement projects.

3. The method for identifying potential hazards in water conservancy construction based on the LDA model according to claim 1, characterized in that: The effectiveness of the topic system was confirmed by comparing the distribution rate and proportion of potential risks in the training and validation samples.

4. The method for identifying potential hazards in water conservancy construction based on the LDA model according to claim 1, characterized in that: It also includes analyzing the distribution characteristics of each hidden danger theme; analyzing the distribution characteristics of each hidden danger theme from the perspectives of individual behavior, equipment and facilities, and organizational management, and clarifying the focus of each hidden danger theme in safety production management; differential analysis and generation of preventive measures; and proposing accident prevention measures from the perspectives of rapid handling of visible hidden dangers and prevention and control of potential risks by comparing the distribution differences of accident causative factors and each hidden danger theme.

5. The method for identifying potential hazards in water conservancy construction based on the LDA model according to claim 1, characterized in that: The formula for calculating the TF-IDF weight value of each word after word segmentation is as follows: In the formula: Vocabulary after word segmentation for hazard investigation record text The weight value; For vocabulary Text dataset of hazard investigation records The number of times in; Text dataset for hazard investigation records The number of times each word appears; For the k-th word; The total number of documents in the text dataset recording hazard investigation records; For words The number of documents.

6. The method for identifying potential hazards in water conservancy construction based on the LDA model according to claim 1, characterized in that: The optimal number of topics is set to the number of topics sampled by Gibbs, with hyperparameters α and β set to auto. The process is iterated several times until Gibbs sampling converges. The conditional probability of a word belonging to each potential topic is calculated by updating the potential topic assignment for each word.

7. The method for identifying potential hazards in water conservancy construction based on the LDA model according to claim 4, characterized in that: The analysis of the distribution characteristics of each potential hazard theme specifically involves statistically analyzing the distribution proportions of individual behavior-related themes, equipment and facility-related themes, and organizational management-related themes, providing a basis for the allocation of safety management resources.

8. The method for identifying potential hazards in water conservancy construction based on the LDA model according to claim 4, characterized in that: The differentiated analysis and prevention measures generation includes comparing the proportion of organizational and management factors in accident causes with the distribution rate of organizational and management themes in hidden danger topics, analyzing the management gap between explicit hidden dangers and potential risks, and proposing optimization measures for the dual prevention mechanism, including constructing a database linking hazard sources and construction sites and a high-frequency hidden danger knowledge base.

Images