Subway construction period risk identification method based on text mining

Abstract

The invention discloses a subway construction period risk identification method based on text mining, and relates to the technical field of engineering risk management. Comprising the following steps: establishing a corpus, and selecting potential safety hazard problem records in a construction period as the corpus; performing data preprocessing on the corpus; the method comprises the following steps of: carrying out word frequency statistics by using a collections.Counter class of Python; carrying out statistics on the high-frequency associated word pairs; drawing a tree diagram based on the high-frequency feature words and the high-frequency associated vocabularies, and displaying a plurality of feature words with the highest word frequency and associated vocabularies; risk features are mined through the obtained high-frequency feature word pairs and tree diagrams of associated vocabularies, and key risk information is confirmed by searching sentences of the feature words in an initial corpus. According to the method, important information of a large amount of text data generated during metro construction is fully mined, so that site construction is better guided, and important support is provided for safety, high efficiency and success of metro construction projects.

Description

Technical Field

[0001] The present invention relates to the technical field of engineering risk management, and more specifically to a method for identifying risks during subway construction based on text mining. Background Art

[0002] The subway construction process involves a wide range of links and variables, and due to industry habits, a large amount of unstructured data will be generated. One of the most important unstructured data is text data, which stores a large amount of important potential information.

[0003] However, the current risk management process mainly focuses on the application of existing subway construction risk databases, especially non-structured data, at the basic data entry, storage, simple query and export levels, but fails to fully explore the important potential information contained in these historical data.

[0004] The present invention extracts information on all potential risks generated during subway construction based on a text mining method, identifies key risks, and thus improves the risk management efficiency during subway construction.

[0005] Therefore, it is an urgent problem for technical personnel in this field to propose a risk identification method for subway construction period based on text mining to solve the difficulties existing in the existing technology. Summary of the invention

[0006] In view of this, the present invention provides a subway construction risk identification method based on text mining, which fully mines the important information of the large amount of text data generated during subway construction, so as to better guide on-site construction and provide important support for the safety, efficiency and success of subway construction projects.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A method for identifying risks during subway construction period based on text mining, comprising the following steps:

[0009] Establish a corpus and select the safety hazard records during the construction period as the corpus;

[0010] Perform data preprocessing on the corpus;

[0011] Use Python's collections.Counter class to count word frequencies;

[0012] Count the high-frequency associated word pairs;

[0013] Draw a tree diagram based on high-frequency feature words and high-frequency related words to display several feature words with the highest word frequency and related words;

[0014] The risk features are mined through the obtained high-frequency feature word pairs and the dendrogram of related words, and the key risk information is confirmed by searching for sentences with feature words in the initial corpus.

[0015] In the above method, the specific content of data preprocessing for the corpus is optional:

[0016] With the help of Jieba word segmentation tool, text sentences are segmented into independent words. Jieba word segmentation tool supports user-defined dictionaries and provides support for Chinese text analysis such as word frequency statistics and stop word processing.

[0017] Combine the corpus content to form a custom word segmentation dictionary. Each word in the custom dictionary text document occupies one line. Use jieba.add_word to add each word in the custom dictionary text document to the jieba dictionary to improve the word segmentation effect.

[0018] Remove stop words.

[0019] The above method optionally uses Python's collections.Counter class to perform word frequency statistics as follows:

[0020] Use Python's collections.Counter class to count the number of occurrences of each word. Counter is a data structure designed for counting. It receives an iterator and creates a dictionary. Then use the most_common method of the Counter object to get and print the most common words and their frequencies. It allows users to specify the number of high-frequency words they want to view, sort them in descending order of frequency, and display the results in the form of a word cloud.

[0021] In the above method, optionally, the specific content of the statistics of high-frequency associated word pairs is:

[0022] The logic of counting related word pairs is implemented by defining a fixed window size. This window slides in the sentence after each word segmentation to capture the word pairs that appear simultaneously within this window.

[0023] When two words appear within 5 words before and after each other, they are considered to be related word pairs. The windows before and after each word are checked to find all the words that co-occur with it. Each pair of words is stored in a certain order to avoid duplication. Every time a co-occurrence relationship is found, the count of the corresponding word pair is added by one. The co-occurrence frequency is converted into another DataFrame with column names 'Word Pair' and 'Co-occurrence', and sorted in descending order by the number of co-occurrences. The results are displayed in the form of a word cloud.

[0024] In the above method, optionally, co-occurrence refers to the situation where two words appear together within a specific window range, and the window is a sentence, a paragraph, or a fixed word range.

[0025] The above method can optionally draw a tree diagram based on high-frequency feature words and high-frequency related words to display the specific contents of several feature words with the highest word frequency and the related words as follows:

[0026] Initialize the graph: Use the networkx library to create a graph G, which is an undirected graph used to store nodes and edges;

[0027] Select and add nodes: Get the most frequent feature words from word_freq_dfDataFrame and add them as nodes to the graph;

[0028] Add edges and associated nodes: For each high-frequency feature word, find the top five word pairs with the highest co-occurrence frequency. For each word pair, add the associated word as a node and add an edge with the weight of the co-occurrence frequency.

[0029] Calculate and set the layout of the graph: Use the spring_layout algorithm to calculate the layout of the graph. This layout keeps connected nodes close and unconnected nodes far away. The layout is optimized by adjusting k, the spring length coefficient, and iterations, the number of iterations.

[0030] It can be seen from the above technical solutions that, compared with the prior art, the present invention discloses a method for identifying risks during subway construction based on text mining, which has the following beneficial effects:

[0031] (1) By adopting text mining technology, it is possible to more efficiently and deeply mine the rich information in unstructured text data and identify key and potential risk factors;

[0032] (2) Through automated text analysis and risk assessment processes, the efficiency and accuracy of risk identification have been significantly improved, which not only shortens the time required for risk assessment, but also improves the timeliness and accuracy of risk warnings, thereby providing more effective support for risk management and decision-making;

[0033] (3) Through the mining of the correlation of characteristic words, not only can risks be identified, but also a scientific basis can be provided for the formulation of risk response measures by analyzing the correlation and impact of risks. This helps the project management team to formulate more effective risk response strategies, optimize resource allocation, and improve construction safety and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0035] Figure 1 A flowchart of a method for identifying risks during subway construction based on text mining provided by the present invention;

[0036] Figure 2 The overall flow chart of a method for identifying risks during subway construction based on text mining provided by the present invention;

[0037] Figure 3 A word cloud diagram of sub-high frequency words provided by an embodiment of the present invention;

[0038] Figure 4 A word cloud diagram of high-frequency word pairs provided by an embodiment of the present invention;

[0039] Figure 5 A tree diagram of high-frequency associated words provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] See also Figure 1 and Figure 2 As shown, the present invention discloses a method for identifying risks during subway construction based on text mining, comprising the following steps:

[0042] Establish a corpus and select the safety hazard records during the construction period as the corpus;

[0043] Perform data preprocessing on the corpus;

[0044] Use Python's collections.Counter class to count word frequencies;

[0045] Count the high-frequency associated word pairs;

[0046] Draw a tree diagram based on high-frequency feature words and high-frequency related words to display several feature words with the highest word frequency and related words;

[0047] The risk features are mined through the obtained high-frequency feature word pairs and the dendrogram of related words, and the key risk information is confirmed by searching for sentences with feature words in the initial corpus.

[0048] Furthermore, the specific contents of data preprocessing for the corpus are as follows:

[0049] With the help of Jieba word segmentation tool, text sentences are segmented into independent words. Jieba word segmentation tool supports user-defined dictionaries and provides support for Chinese text analysis such as word frequency statistics and stop word processing.

[0050] Specifically, Jieba word segmentation tool supports user-defined dictionaries and has the advantages of high efficiency, accuracy, flexibility, ease of use and scalability.

[0051] Combine the corpus content to form a custom word segmentation dictionary. Each word in the custom dictionary text document occupies one line. Use jieba.add_word to add each word in the custom dictionary text document to the jieba dictionary to improve the word segmentation effect.

[0052] Specifically, a custom word segmentation dictionary is formed by combining the corpus content, such as fixed expressions, professional terms for subway construction, etc.

[0053] Remove stop words.

[0054] Furthermore, the specific contents of using Python's collections.Counter class for word frequency statistics are as follows:

[0055] Use Python's collections.Counter class to count the number of occurrences of each word. Counter is a data structure designed for counting. It receives an iterator and creates a dictionary. Then use the most_common method of the Counter object to get and print the most common words and their frequencies. It allows users to specify the number of high-frequency words they want to view, sort them in descending order of frequency, and display the results in the form of a word cloud.

[0056] Specifically, for example, to display the top 20 words with the highest number of occurrences, convert the total word frequency into a DataFrame with columns named 'Word' and 'Frequency', sort them in descending order by word frequency, and display the results in a word cloud.

[0057] Furthermore, the specific contents of the statistics of high-frequency associated word pairs are as follows:

[0058] The logic of counting related word pairs is implemented by defining a fixed window size. This window slides in the sentence after each word segmentation to capture the word pairs that appear simultaneously within this window.

[0059] When two words appear within 5 words before and after each other, they are considered to be related word pairs. The windows before and after each word are checked to find all the words that co-occur with it. Each pair of words is stored in a certain order to avoid duplication. Every time a co-occurrence relationship is found, the count of the corresponding word pair is added by one. The co-occurrence frequency is converted into another DataFrame with column names 'Word Pair' and 'Co-occurrence', and sorted in descending order by the number of co-occurrences. The results are displayed in the form of a word cloud.

[0060] Specifically, suppose the window size is set to 5, and a sentence is "A / B / C / D / E / F / G / H / I / J" (one letter represents one word) after preprocessing. For C, forward search, the associated word pairs are (B, C), (A, C), and backward search, the associated word pairs are (C, D), (C, E), (C, F), (C, G), (C, H). Specifically, when two words appear within 5 words before and after each other, they are considered to be associated word pairs.

[0061] Furthermore, co-occurrence refers to the situation where two words appear together within a specific window range, which is a sentence, a paragraph, or a fixed word range.

[0062] Furthermore, a tree diagram is drawn based on high-frequency feature words and high-frequency related words, showing the specific contents of several feature words with the highest word frequency and the related words as follows:

[0063] Initialize the graph: Use the networkx library to create a graph G, which is an undirected graph used to store nodes and edges;

[0064] Select and add nodes: Get the most frequent feature words from word_freq_dfDataFrame and add them as nodes to the graph;

[0065] Add edges and associated nodes: For each high-frequency feature word, find the top five word pairs with the highest co-occurrence frequency. For each word pair, add the associated word as a node and add an edge with the weight of the co-occurrence frequency.

[0066] Calculate and set the layout of the graph: Use the spring_layout algorithm to calculate the layout of the graph. This layout keeps connected nodes close and unconnected nodes far away. The layout is optimized by adjusting k, the spring length coefficient, and iterations, the number of iterations.

[0067] In a specific embodiment:

[0068] Step 1: Save 4057 potential problems recorded in a certain period of subway construction as a text document, with each record in the document as a line, and use it as a corpus;

[0069] Step 2: Data preprocessing;

[0070] Step 2.1, create a custom word segmentation dictionary, add each word in the custom dictionary document to the Jieba dictionary to improve the word segmentation effect; the custom word segmentation dictionary includes fixed expressions, subway construction special words, etc., such as: "Construction plan", "Emergency management", "Video surveillance", "Emergency materials", "Grouting record", "Briefing record", "Construction record", "Contact channel", "Operation platform", "Safety management", "Grouting pressure", "Construction operation", "Monitoring plan", "Secondary grouting", "Duty record", "Operator", "Technical briefing", "Edge protection", etc.;

[0071] Step 2.2, establish a stop word list, such as: "year month day", "schedule", "station", "interval", "not", "supervision", "level", "and", "but", "and", "remind", "law", "agreement", "before", "after", "part", etc.;

[0072] Step 2.3, use the word segmentation tool Jieba to split the text sentences into independent words and remove the words in the stop word list;

[0073] Step 3: Use Python's collections.Counter class to count the number of occurrences of each word and display the results in the form of a word cloud, so as to intuitively obtain the initial risk characteristics. Figure 3 As shown, it can be preliminarily determined that the locations that need to be focused on include: tunnels, entrances and exits, foundation pits, monitoring points, passages, ground, etc., and the categories that need to be focused on include: equipment, steel bars, fire protection (fire extinguishers), video surveillance (cameras), emergency, monitoring, etc.

[0074] Step 4: When two words appear within 5 words before and after each other, they are considered to be related word pairs. Check the windows before and after each word, find all the words that co-occur with it, and store each pair of words (word pairs) in a certain order (such as dictionary order) to avoid duplication. Display the co-occurring high-frequency phrases in the form of word clouds, so as to further obtain key feature risks, such as Figure 4 As shown;

[0075] Step 5: Draw a tree diagram using the high-frequency feature words and related words to display the 15 most frequent feature words and their related words, and further identify the key risks that need attention, such as Figure 5 As shown;

[0076] Step 6: By searching for sentences with high-frequency feature words and related words in the initial corpus, we can confirm that the key risk information is: monitoring issues, foundation pit excavation issues, video surveillance issues, operator issues, and emergency issues are more prominent, specifically:

[0077] (1) The camera is not aimed at or does not cover the work surface;

[0078] (2) The video surveillance camera did not follow the excavation surface in a timely manner;

[0079] (3) Some areas are not equipped with video surveillance cameras;

[0080] (4) Monitoring points are not fully deployed;

[0081] (5) The third-party monitoring unit fails to perform its duties properly;

[0082] (6) There are problems in the comparative analysis of monitoring data;

[0083] (7) The acceptance records of the construction unit are incomplete;

[0084] (8) Some areas are not equipped with edge protection;

[0085] (9) Water accumulation in foundation pits, tunnels, underground mines and other areas is not promptly pumped out;

[0086] (10) The drainage ditch was not cleaned in time.

[0087] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0088] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for identifying risks during subway construction based on text mining, characterized in that: The following steps are involved: Establish a corpus and select the safety hazard records during the construction period as the corpus; Perform data preprocessing on the corpus; Use Python's collections.Counter class to count word frequencies; Count the high-frequency associated word pairs; Draw a tree diagram based on high-frequency feature words and high-frequency related words to display several feature words with the highest word frequency and related words; The risk features are mined through the obtained high-frequency feature word pairs and the dendrogram of related words, and the key risk information is confirmed by searching for sentences with feature words in the initial corpus.

2. According to claim 1, a method for identifying risks during subway construction based on text mining is characterized in that: The specific contents of data preprocessing for the corpus are as follows: With the help of Jieba word segmentation tool, text sentences are segmented into independent words. Jieba word segmentation tool supports user-defined dictionaries and provides support for Chinese text analysis such as word frequency statistics and stop word processing. Combine the corpus content to form a custom word segmentation dictionary. Each word in the custom dictionary text document occupies one line. Use jieba.add_word to add each word in the custom dictionary text document to the jieba dictionary to improve the word segmentation effect. Remove stop words.

3. The method for identifying risks during subway construction period based on text mining according to claim 1 is characterized in that: The specific contents of using Python's collections.Counter class for word frequency statistics are as follows: Use Python's collections.Counter class to count the number of occurrences of each word. Counter is a data structure designed for counting. It receives an iterator and creates a dictionary. Then use the most_common method of the Counter object to get and print the most common words and their frequencies. It allows users to specify the number of high-frequency words they want to view, sort them in descending order of frequency, and display the results in the form of a word cloud.

4. The method for identifying risks during subway construction period based on text mining according to claim 1 is characterized in that: The specific contents of statistics on high-frequency associated word pairs are as follows: The logic of counting related word pairs is implemented by defining a fixed window size. This window slides in the sentence after each word segmentation to capture the word pairs that appear simultaneously within this window. When two words appear within 5 words before and after each other, they are considered to be related word pairs. The windows before and after each word are checked to find all the words that co-occur with it. Each pair of words is stored in a certain order to avoid duplication. Every time a co-occurrence relationship is found, the count of the corresponding word pair is added by one. The co-occurrence frequency is converted into another DataFrame with column names 'Word Pair' and 'Co-occurrence', and sorted in descending order by the number of co-occurrences. The results are displayed in the form of a word cloud.

5. The method for identifying risks during subway construction period based on text mining according to claim 4 is characterized in that: Co-occurrence refers to the situation where two words appear together within a specific window, which can be a sentence, a paragraph, or a fixed range of words.

6. The method for identifying risks during subway construction period based on text mining according to claim 1 is characterized in that: Based on the high-frequency feature words and high-frequency related words, a tree diagram is drawn to show the specific contents of several feature words with the highest word frequency and the related words: Initialize the graph: Use the networkx library to create a graph G, which is an undirected graph used to store nodes and edges; Select and add nodes: Get the most frequent feature words from word_freq_dfDataFrame and add them as nodes to the graph; Add edges and associated nodes: For each high-frequency feature word, find the top five word pairs with the highest co-occurrence frequency. For each word pair, add the associated word as a node and add an edge with the weight of the co-occurrence frequency. Calculate and set the layout of the graph: Use the spring_layout algorithm to calculate the layout of the graph. This layout keeps connected nodes close and unconnected nodes far away. The layout is optimized by adjusting k, the spring length coefficient, and iterations, the number of iterations.

Images