Coal mine water disaster early warning method based on Spark
By introducing Spark parallel computing framework and support vector machine model, the problems of data processing efficiency and accuracy in coal mine water damage warning are solved, and efficient online warning of coal mine water damage is achieved.
Patent Information
- Application Number
- CN202411117664.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-15
- Publication Date
- 2024-12-27
AI Technical Summary
The existing coal mine water damage early warning methods are difficult to effectively analyze and utilize large-scale water damage data, and the traditional methods have limited accuracy in predicting coal mine water damage, and cannot quickly process large amounts of data, resulting in difficulty in identifying and early warning of coal mine water damage risks.
The Spark parallel computing framework is adopted to select feature through the random forest regression algorithm, build a support vector machine prediction model, and use Spark Streaming to complete streaming calculation and storage, realizing the construction of an online early warning model for coal mine water damage.
It improves the overall accuracy and timeliness of coal mine water damage warning, avoids overfitting during model training, can quickly process large amounts of data, and provides real-time early warning.
Smart Images
Figure CN119207044A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of coal mine safety, and in particular to a coal mine water hazard early warning method based on Spark. Background Art
[0002] my country's coal mining method is mainly underground mining. The underground environment of coal mines is extremely complex. In addition, most coal mines have low safety management levels and poor prevention awareness, which leads to frequent coal mine safety accidents. In the process of mine construction and production, coal mine water hazards are one of the main safety disasters along with gas and roof. Therefore, achieving effective early warning of coal mine water hazards is crucial to ensuring safe production in coal mines.
[0003] In the process of digital mine development, coal mine safety monitoring technology and automated data collection methods have been applied in most coal mines, which have improved the safety level of coal mines to a certain extent, reduced the occurrence of safety accidents such as coal mine water inrush, and provided a large amount of raw data for coal mine water inrush research. However, these mine water hazard data are complex and difficult to use effectively. How to analyze and use large-scale water hazard data has become an urgent need for coal mine water prevention and control.
[0004] Traditional water hazard warning methods often issue warnings by detecting whether a single factor exceeds a threshold, while the formation of coal mine water hazards is affected by many factors, such as aquifer thickness, water pressure, aquiclude thickness, burial depth, distance from the working face, etc. It is difficult to give a definite conclusion using a discriminant formula alone, and the accuracy of coal mine water hazard prediction is limited. In addition, traditional coal mine water hazard warnings mostly use manual discrimination methods or water hazard coefficient methods, which require a high level of experts and are difficult to quickly process the large amount of data generated in digital mines, and cannot quickly discriminate and warn of the current water hazard risks in coal mines. Summary of the invention
[0005] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to propose a coal mine water hazard early warning method based on Spark. By introducing the Spark parallel computing framework, the coal mine water hazard early warning research is carried out, the support vector machine (SVM) prediction model is loaded using PMML, and the streaming computing and storage design is completed using Spark Streaming, thereby realizing the construction of an online early warning model for coal mine water hazard. Compared with the stand-alone method, the overall accuracy and timeliness of the coal mine water hazard early warning are effectively improved.
[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0007] A coal mine water hazard early warning method based on Spark, the method comprising:
[0008] Collect and organize the original data of coal mine water hazards and generate the original data set of coal mine water hazards;
[0009] The random forest regression algorithm is used to select features of the original data set of coal mine water damage to generate a data set of coal mine water damage;
[0010] Preprocess the coal mine water hazard dataset, divide the coal mine water hazard dataset into a training set and a test set, and build a coal mine water hazard prediction model;
[0011] The training set is used to train the coal mine water hazard prediction model, and the test set is used to predict the trained coal mine water hazard prediction model;
[0012] Build a cloud computing cluster, load the trained coal mine water hazard prediction model into the cloud computing framework, and perform coal mine water hazard data stream computing design to obtain an online early warning model for coal mine water hazard;
[0013] The coal mine water hazard online early warning model is used to receive and predict the real-time data of coal mine water hazard.
[0014] The raw data of coal mine water hazards are various characteristic data related to coal mine water hazards, including aquifer thickness, water pressure, aquiclude thickness, sand-rock ratio, mud-rock ratio, elevation, coal seam inclination, fault drop, distance from fault, mining height, mining depth and mining speed.
[0015] The random forest regression algorithm is used to perform feature selection on the original data set of coal mine water damage to generate a data set of coal mine water damage, including:
[0016] Randomly select some feature data in the original data set of coal mine water hazards to construct a random forest, wherein the random forest includes a number of decision trees;
[0017] For each decision tree, out-of-bag data is used to calculate out-of-bag data error, where the out-of-bag data refers to some feature data in the original data set of coal mine water hazards that are not involved in establishing the decision tree;
[0018] Add noise to the out-of-bag data and repeatedly calculate the out-of-bag data error;
[0019] Integrate the out-of-bag data errors of several decision trees to obtain the influence weight of the feature data;
[0020] The importance of feature data is judged according to the weight of its influence. If the accuracy of out-of-bag data decreases after noise is added, the feature data has a greater impact on the result, indicating that the feature data is more important.
[0021] The coal mine water hazard dataset is constructed by selecting the top five feature data that have the greatest impact on the results from the original coal mine water hazard dataset.
[0022] The coal mine water hazard dataset is preprocessed to divide the coal mine water hazard dataset into a training set and a test set, including:
[0023] The coal mine water hazard data set is standardized by using a minimum-maximum standardization formula, and the sample data is mapped to the interval [0,1] to obtain a standardized coal mine water hazard data set;
[0024] The standardized coal mine water hazard data set is divided into a training set and a test set using the random sampling method;
[0025] The training set is divided into K non-overlapping subsets through the K-fold cross-validation method. In each iteration, the 1st to K-1th groups of data are used for model training, and the Kth group of data is used for verification.
[0026] The coal mine water hazard prediction model uses a support vector machine model, selects an RBF kernel function as a model kernel function, and adopts a grid search algorithm to select the optimal ranges of two parameters, an error penalty function C and a kernel function g.
[0027] The cloud computing cluster is constructed, the trained coal mine water hazard prediction model is loaded into the cloud computing framework, and the coal mine water hazard data stream computing design is performed to obtain the coal mine water hazard online early warning model, including:
[0028] Use the Spark on YARN mode to deploy a Spark cluster and perform cluster planning, set up a master node and several slave nodes, and obtain the Spark framework;
[0029] The trained coal mine water hazard prediction model is persisted in PMML format to obtain a persistent coal mine water hazard prediction model, and loaded into the Spark framework;
[0030] Spark Streaming is used as the receiving and processing method of coal mine water hazard data stream. The batch processing time interval is set. The coal mine water hazard data stream within each batch processing time interval is an RDD. Several RDDs form a DStream. The coal mine water hazard data stream is sent using TCP Socket, and Spark Streaming receives the coal mine water hazard data stream to obtain the coal mine water hazard online early warning model.
[0031] The method of receiving and predicting the real-time data of coal mine water hazards by using the coal mine water hazard online early warning model includes:
[0032] Collect real-time data on coal mine water hazards and input it into the online early warning model for coal mine water hazards;
[0033] Convert the real-time data stream of coal mine water damage into DStream format to obtain the real-time data stream of coal mine water damage;
[0034] The TCP Socket in the coal mine water hazard online warning model divides the coal mine water hazard real-time data stream into several DStreams according to the set batch processing time interval, and each DStream includes several RDDs;
[0035] TCP Socket sends DStream to Spark Streaming at time intervals;
[0036] Spark Streaming inputs DStream into the persistent coal mine water disaster prediction model;
[0037] The persistent coal mine water disaster prediction model predicts each RDD in the DStream and obtains the prediction results;
[0038] The prediction results are stored in the database to complete the early warning of coal mine water hazards.
[0039] A coal mine water hazard early warning system based on Spark, comprising:
[0040] Data collection module, collects and organizes the original data of coal mine water damage and generates the original data set of coal mine water damage;
[0041] The feature selection module uses the random forest regression algorithm to perform feature selection on the original data set of coal mine water damage to generate a data set of coal mine water damage;
[0042] The coal mine water hazard prediction model building module preprocesses the coal mine water hazard data set, divides the coal mine water hazard data set into a training set and a test set, and builds a coal mine water hazard prediction model;
[0043] Model training module, using the training set to train the coal mine water hazard prediction model, and using the test set to predict the trained coal mine water hazard prediction model;
[0044] The coal mine water hazard online early warning model construction module builds a cloud computing cluster, loads the trained coal mine water hazard prediction model into the cloud computing framework, and performs coal mine water hazard data stream computing design to obtain the coal mine water hazard online early warning model;
[0045] The data prediction module uses the coal mine water hazard online early warning model to receive and predict the real-time data of coal mine water hazard.
[0046] An electronic device, comprising:
[0047] at least one processor;
[0048] and, a memory communicatively coupled to the at least one processor;
[0049] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned coal mine water hazard early warning methods based on Spark.
[0050] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the Spark-based coal mine water hazard early warning method described in any one of the above items is implemented.
[0051] Compared with the prior art, the beneficial effects of the present invention are:
[0052] 1. The Spark-based coal mine water hazard early warning method provided by the present invention adopts a random forest algorithm to screen the sample features in the original data set of coal water hazard, retains the features with a large correlation with the prediction results, avoids the occurrence of overfitting during the model training process, and improves the model training efficiency.
[0053] 2. The Spark-based coal mine water hazard early warning method provided by the present invention adopts a support vector machine algorithm to predict coal mine water hazards, normalizes the coal mine water hazard data, uses the K-fold cross-validation method to train the processed data, optimizes the model parameters using a grid search algorithm, selects the kernel function, experimentally verifies and analyzes the model, and improves the overall accuracy of the coal mine water hazard prediction model.
[0054] In summary, the present invention aims at the problem of online processing of coal mine data, introduces the Spark parallel computing framework to conduct research on coal mine water hazard early warning, uses PMML to load the support vector machine (SVM) prediction model, and uses Spark Streaming to complete the design of streaming computing and storage, thereby realizing the construction of an online early warning model for coal mine water hazard. Compared with the stand-alone method, the overall accuracy and timeliness of coal mine water hazard early warning are effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] Figure 1 The figure is a flow chart of the method of the present invention.
[0056] Figure 2 This is a feature correlation histogram of the random forest algorithm of the present invention.
[0057] Figure 3 It is the fitness grid diagram of the grid search algorithm of the present invention.
[0058] Figure 4 This is a schematic diagram of the persistence principle of the coal mine water hazard prediction model of the present invention.
[0059] Figure 5This is a line graph comparing the running time of the Spark-based coal mine water hazard early warning method described in the present invention and the traditional single-machine coal mine water hazard early warning method. DETAILED DESCRIPTION
[0060] The technical solution of the present invention is described in detail below with reference to the accompanying drawings and embodiments.
[0061] A coal mine water disaster early warning method based on Spark, such as Figure 1 As shown, the method includes:
[0062] S1, collect and organize the original data of coal mine water hazards to generate the original data set of coal mine water hazards;
[0063] The raw data of coal mine water hazards are various characteristic data related to coal mine water hazards, including aquifer thickness, water pressure, aquiclude thickness, sand-rock ratio, mud-rock ratio, elevation, coal seam inclination, fault drop, distance from fault, mining height, mining depth and mining speed, and whether there is water inrush. The sample data should contain multiple characteristic values, and the data distribution should be uniform.
[0064] S2, using random forest regression algorithm to select features of the original data set of coal mine water damage, to generate a data set of coal mine water damage, specifically:
[0065] Randomly select some feature data in the original data set of coal mine water hazards to construct a random forest, wherein the random forest includes N decision trees;
[0066] For each decision tree, the out-of-bag data (OOB) is used to calculate the out-of-bag data error, denoted as eOOB1. The out-of-bag data refers to some feature data in the original data set of coal mine water hazards that are not involved in establishing the decision tree. These data can be used to evaluate the performance of the decision tree.
[0067] Add noise to the out-of-bag data and repeatedly calculate the out-of-bag data error, denoted as eOOB2;
[0068] Integrate the out-of-bag data errors of N decision trees to obtain the influence weight of feature X ∑(eOOB2-eOOB1) / N. The feature correlation results obtained by the random forest algorithm are as follows Figure 2 As shown;
[0069] The importance of feature data is judged according to the weight of its influence. If the accuracy of out-of-bag data decreases (eOOB2 increases) after noise is added, the feature data has a greater impact on the result, indicating that the feature data is more important.
[0070] The coal mine water hazard dataset is constructed by selecting the top five feature data that have the greatest impact on the results from the original coal mine water hazard dataset.
[0071] S3, preprocess the coal mine water hazard dataset, divide the coal mine water hazard dataset into a training set and a test set, and use the support vector machine model to build a coal mine water hazard prediction model, specifically:
[0072] The coal mine water hazard data set is standardized by using the minimum-maximum standardization formula, and the sample data is mapped to the [0,1] interval to obtain the standardized coal mine water hazard data set. The minimum-maximum standardization calculation formula is as follows:
[0073]
[0074] Where: x n is the data obtained after standardization, x is the original data, and x max is the maximum data, x min is the minimum data;
[0075] The standardized coal mine water hazard data set is divided into a training set and a test set using a random sampling method, with the percentages of 80% and 20% respectively.
[0076] The training set is divided into K non-overlapping subsets by the K-fold cross-validation method. In each iteration, the 1st to K-1th groups of data are used for model training, and the Kth group of data is used for validation.
[0077] The coal mine water hazard prediction model uses a support vector machine model, selects the RBF kernel function as the model kernel function, and uses a grid search algorithm to select the optimal range of the two parameters of the error penalty function C and the kernel function g, specifically:
[0078] Grid search combines different values of C and g into a two-dimensional grid. Each independent cell in the grid represents a parameter pair (C, g). The optimal parameter combination can be found by traversing all parameter pairs. The steps of using the grid search algorithm include dividing the search range, cross-validating the grid classification accuracy, selecting the optimal parameters, and narrowing the search range.
[0079] At the beginning of the grid search, the initial search range is set to 2-10≤C≤210, 2-10≤g≤210. The np.logspace(start,stop,num,base) method is used to construct a geometric sequence with base as the base and a length of num from the start value to the end value. This method is used to take 100 numbers in the respective ranges of C and g to form a grid. Using 3-fold cross validation, the accuracy of each group of C and g values in the search range is plotted on a three-dimensional grid graph. The resulting grid graph is the fitness grid graph of the grid search algorithm during the training process, as shown in the figure below. Figure 3 shown.
[0080] S4, using the training set to train the coal mine water hazard prediction model, using the test set to predict the trained coal mine water hazard prediction model, and evaluating the prediction results, specifically:
[0081] Based on the divided training set, the coal mine water hazard prediction model is trained;
[0082] Based on the divided test set, the prediction effect of the coal mine water hazard prediction model was tested, and the model effect was evaluated according to four evaluation indicators: accuracy, precision, recall, and F1 value.
[0083] S5, build a cloud computing cluster, load the trained coal mine water hazard prediction model into the cloud computing framework, and perform coal mine water hazard data stream computing design to obtain a coal mine water hazard online early warning model, including:
[0084] Use the Spark on YARN mode to deploy a Spark cluster and perform cluster planning, set up a master node and several slave nodes, and obtain the Spark framework;
[0085] The trained coal mine water hazard prediction model is persisted in PMML format to obtain a persistent coal mine water hazard prediction model, and loaded into the Spark framework;
[0086] Spark Streaming is used as the receiving and processing method of coal mine water hazard data stream. The batch processing time interval is set. The coal mine water hazard data stream within each batch processing time interval is an RDD. Several RDDs form a DStream. The coal mine water hazard data stream is sent using TCP Socket, and Spark Streaming receives the coal mine water hazard data stream to obtain the coal mine water hazard online early warning model.
[0087] Specifically:
[0088] Use Spark on YARN mode to deploy Spark clusters, and plan clusters according to application scenarios; one node is the master node (Master), and the rest are slave nodes (Slave). Run Hadoop's NameNode, SecondaryNameNode, DataNode, ResourceManager and other processes, Spark's Master and Worker processes in the master node; and run Hadoop's DataNode and NodeManager, Spark's Worker process in the slave nodes;
[0089] The coal mine water hazard prediction model is persisted and loaded into the Spark framework, and PMML is used to persist and load the model; specifically, the constructed coal mine water hazard warning model is persisted into PMML format through the JPMML-Sklearn model conversion library, and then loaded into the Spark framework through the JPMML-Evaluator model conversion library;
[0090] The coal mine water hazard data streaming computing design is carried out, and Spark Streaming is used as the receiving and processing method of streaming data; first, the stream processing engine is started, and the batch processing time interval is set to 1 second; then Spark Streaming is used to continuously receive the streaming data sent by TCP Socket, and form batch data at a certain interval. These batch data correspond to an RDD instance in Spark, and a group of RDDs will form a DStream (discrete stream).
[0091] In one embodiment, the Spark on YARN mode is first used to deploy the Spark cluster, and three nodes are used to build the Spark cluster, one of which is the master node (Master), and the other two are slave nodes (Slave). The master node runs Hadoop's NameNode, SecondaryNameNode, DataNode, ResourceManager and other processes, and Spark's Master and Worker processes; and the slave nodes run Hadoop's DataNode and NodeManager, and Spark's Worker processes.
[0092] Subsequently, PMML is used to persist the coal mine water hazard prediction model and load it into the Spark framework. The specific steps include: (1) establishing an input stream and reading a persistent file; (2) calling the unmarshal method to realize the conversion of the persistent model to the model object; (3) creating a model factory ModelEvaluatorFactory; (4) the model factory generates a ModelInvoker model instance; the specific principle is as follows Figure 4 shown.
[0093] Finally, Spark Streaming is used as the receiving and processing method for streaming data. First, start the stream processing engine and set the batch processing interval to 1 second; then use Spark Streaming to continuously receive the streaming data sent by TCP Socket and form batch data at a certain interval. These batches of data correspond to an RDD instance in Spark, and a group of RDDs will form a DStream (discrete stream).
[0094] After Spark Streaming calculates the data, the results need to be stored. Here, the Spark SQL component is used to implement persistence operations in the database. Spark SQL can be seamlessly integrated with Spark Streaming to implement batch processing, stream processing, and interactive query of data in one technology stack. The process of using Spark SQL to store processes includes: using config to load the configuration items in Spark Config and create SparkSession; configuring MySQL database connection properties; converting the prediction data output in list form into Dataset form, where each line in the list is warning data saved in String type, including warning information and warning results. Convert non-json format RDD into DataFrame through reflection, and then save it in Dataset form; finally, use Append write operation to write the prediction data to the database.
[0095] S6, using the coal mine water hazard online early warning model to receive and predict the real-time data of coal mine water hazard, including:
[0096] Collect real-time data on coal mine water hazards and input it into the online early warning model for coal mine water hazards;
[0097] Convert the real-time data stream of coal mine water damage into DStream format to obtain the real-time data stream of coal mine water damage;
[0098] The TCP Socket in the coal mine water hazard online warning model divides the coal mine water hazard real-time data stream into several DStreams according to the set batch processing time interval, and each DStream includes several RDDs;
[0099] TCP Socket sends DStream to Spark Streaming at time intervals;
[0100] Spark Streaming inputs DStream into the persistent coal mine water disaster prediction model;
[0101] The persistent coal mine water disaster prediction model predicts each RDD in the DStream and obtains the prediction results;
[0102] The prediction results are stored in the database to complete the early warning of coal mine water hazards.
[0103] Specifically:
[0104] After the received data is converted into the DStream format supported by Streaming, the persisted coal mine water hazard prediction model is used to predict each RDD in the DStream to obtain the prediction results.
[0105] Select the Spark SQL component to implement persistent storage of prediction results, write the prediction results into the database, and ensure that data files can be appended to the data table to avoid omissions or overwriting.
[0106] A coal mine water hazard early warning system based on Spark, comprising:
[0107] Data collection module, collects and organizes the original data of coal mine water damage and generates the original data set of coal mine water damage;
[0108] The feature selection module uses the random forest regression algorithm to perform feature selection on the original data set of coal mine water damage to generate a data set of coal mine water damage;
[0109] The coal mine water hazard prediction model building module preprocesses the coal mine water hazard data set, divides the coal mine water hazard data set into a training set and a test set, and builds a coal mine water hazard prediction model;
[0110] Model training module, using the training set to train the coal mine water hazard prediction model, and using the test set to predict the trained coal mine water hazard prediction model;
[0111] The coal mine water hazard online early warning model construction module builds a cloud computing cluster, loads the trained coal mine water hazard prediction model into the cloud computing framework, and performs coal mine water hazard data stream computing design to obtain the coal mine water hazard online early warning model;
[0112] The data prediction module uses the coal mine water hazard online early warning model to receive and predict the real-time data of coal mine water hazard.
[0113] An electronic device, comprising:
[0114] at least one processor;
[0115] and, a memory communicatively coupled to the at least one processor;
[0116] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of the above-mentioned coal mine water hazard early warning methods based on Spark.
[0117] A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the Spark-based coal mine water hazard early warning method described in any one of the above is implemented.
[0118] Figure 5The comparison of the running time of the coal mine water hazard early warning method based on Spark described in the present invention and the traditional stand-alone coal mine water hazard early warning is shown. It can be seen that when the data scale is small, the time difference between the stand-alone and parallel methods is not large, and the processing speed is basically equal. Even when the amount of data is very small, the efficiency is slightly lower than the stand-alone operation mode. This is because when the amount of data is small, the Spark cluster takes up part of the time in allocating resources and scheduling tasks. When the amount of data increases to a certain level, the processing time of a single machine increases exponentially, while the processing time of the parallel method increases slowly. Therefore, for big data early warning, the efficiency of Spark parallel computing is much higher than that of the stand-alone method.
[0119] In summary, the present invention aims at the problem of online processing of coal mine data, introduces the Spark parallel computing framework to conduct research on coal mine water hazard early warning, loads the SVM prediction model by using PMML, and uses Spark Streaming to complete the design of streaming computing and storage, thereby realizing the construction of an online early warning model for coal mine water hazard. Compared with the stand-alone method, the overall accuracy and timeliness of coal mine water hazard early warning are effectively improved.
[0120] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present application.
Claims
1. A coal mine water disaster early warning method based on Spark, characterized in that: The method comprises: Collect and organize the original data of coal mine water hazards and generate the original data set of coal mine water hazards; The random forest regression algorithm is used to select features of the original data set of coal mine water damage to generate a data set of coal mine water damage; Preprocess the coal mine water hazard dataset, divide the coal mine water hazard dataset into a training set and a test set, and build a coal mine water hazard prediction model; The training set is used to train the coal mine water hazard prediction model, and the test set is used to predict the trained coal mine water hazard prediction model; Build a cloud computing cluster, load the trained coal mine water hazard prediction model into the cloud computing framework, and perform coal mine water hazard data stream computing design to obtain an online early warning model for coal mine water hazard; The coal mine water hazard online early warning model is used to receive and predict the real-time data of coal mine water hazard.
2. The coal mine water hazard early warning method based on Spark according to claim 1 is characterized in that: The raw data of coal mine water hazards are various characteristic data related to coal mine water hazards, including aquifer thickness, water pressure, aquiclude thickness, sand-rock ratio, mud-rock ratio, elevation, coal seam inclination, fault drop, distance from fault, mining height, mining depth and mining speed.
3. The coal mine water hazard early warning method based on Spark according to claim 2 is characterized in that: The random forest regression algorithm is used to perform feature selection on the original data set of coal mine water damage to generate a data set of coal mine water damage, including: Randomly select some feature data in the original data set of coal mine water hazards to construct a random forest, wherein the random forest includes a number of decision trees; For each decision tree, out-of-bag data is used to calculate out-of-bag data error, where the out-of-bag data refers to some feature data in the original data set of coal mine water hazards that are not involved in establishing the decision tree; Add noise to the out-of-bag data and repeatedly calculate the out-of-bag data error; Integrate the out-of-bag data errors of several decision trees to obtain the influence weight of the feature data; The importance of feature data is judged according to the weight of its influence. If the accuracy of out-of-bag data decreases after noise is added, the feature data has a greater impact on the result, indicating that the feature data is more important. The coal mine water hazard dataset is constructed by selecting the top five feature data that have the greatest impact on the results from the original coal mine water hazard dataset.
4. The coal mine water hazard early warning method based on Spark according to claim 3 is characterized in that: The coal mine water hazard dataset is preprocessed to divide the coal mine water hazard dataset into a training set and a test set, including: The coal mine water hazard data set is standardized by using a minimum-maximum standardization formula, and the sample data is mapped to the interval [0,1] to obtain a standardized coal mine water hazard data set; The standardized coal mine water hazard data set is divided into a training set and a test set using the random sampling method; The training set is divided into K non-overlapping subsets through the K-fold cross-validation method. In each iteration, the 1st to K-1th groups of data are used for model training, and the Kth group of data is used for verification.
5. The Spark-based coal mine water hazard early warning method according to claim 4 is characterized in that: The coal mine water hazard prediction model uses a support vector machine model, selects an RBF kernel function as a model kernel function, and adopts a grid search algorithm to select the optimal ranges of two parameters, an error penalty function C and a kernel function g.
6. The Spark-based coal mine water hazard early warning method according to claim 5 is characterized in that: The cloud computing cluster is constructed, the trained coal mine water hazard prediction model is loaded into the cloud computing framework, and the coal mine water hazard data stream computing design is performed to obtain the coal mine water hazard online early warning model, including: Use the Spark on YARN mode to deploy a Spark cluster and perform cluster planning, set up a master node and several slave nodes, and obtain the Spark framework; The trained coal mine water hazard prediction model is persisted in PMML format to obtain a persistent coal mine water hazard prediction model, and loaded into the Spark framework; Spark Streaming is used as the receiving and processing method of coal mine water hazard data stream. The batch processing time interval is set. The coal mine water hazard data stream within each batch processing time interval is an RDD. Several RDDs form a DStream. TCPSocket is used to send the coal mine water hazard data stream, and Spark Streaming receives the coal mine water hazard data stream to obtain the coal mine water hazard online early warning model.
7. The Spark-based coal mine water hazard early warning method according to claim 6 is characterized in that: The method of receiving and predicting the real-time data of coal mine water hazards by using the coal mine water hazard online early warning model includes: Collect real-time data on coal mine water hazards and input it into the online early warning model for coal mine water hazards; Convert the real-time data stream of coal mine water damage into DStream format to obtain the real-time data stream of coal mine water damage; The TCP Socket in the coal mine water hazard online warning model divides the coal mine water hazard real-time data stream into several DStreams according to the set batch processing time interval, and each DStream includes several RDDs; TCP Socket sends DStream to Spark Streaming at time intervals; Spark Streaming inputs DStream into the persistent coal mine water disaster prediction model; The persistent coal mine water disaster prediction model predicts each RDD in the DStream and obtains the prediction results; The prediction results are stored in the database to complete the early warning of coal mine water hazards.
8. A coal mine water hazard early warning system based on Spark, characterized in that: include: Data collection module, collects and organizes the original data of coal mine water damage and generates the original data set of coal mine water damage; The feature selection module uses the random forest regression algorithm to perform feature selection on the original data set of coal mine water damage to generate a data set of coal mine water damage; The coal mine water hazard prediction model construction module preprocesses the coal mine water hazard data set, divides the coal mine water hazard data set into a training set and a test set, and constructs a coal mine water hazard prediction model; Model training module, using the training set to train the coal mine water hazard prediction model, and using the test set to predict the trained coal mine water hazard prediction model; The coal mine water hazard online early warning model construction module builds a cloud computing cluster, loads the trained coal mine water hazard prediction model into the cloud computing framework, and performs coal mine water hazard data stream computing design to obtain the coal mine water hazard online early warning model; The data prediction module uses the coal mine water hazard online early warning model to receive and predict the real-time data of coal mine water hazard.
9. An electronic device, characterized in that: include: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the Spark-based coal mine water hazard early warning method as described in any one of claims 1 to 7.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the Spark-based coal mine water hazard early warning method as described in any one of claims 1 to 7 is implemented.