Gas concentration real-time prediction method driven by Spark Streaming framework

Abstract

The invention discloses a gas concentration real-time prediction method driven by a Spark Streaming framework, and the method comprises the steps: obtaining real-time gas concentration data, carrying out the preprocessing of the gas data through Spark Streaming, obtaining flow data, and dividing the flow data into a training set and a test set according to the sequence; constructing an ARIMA-SVM gas concentration prediction model, and inputting the training set into the prediction model to train the prediction model; inputting test set data into the trained prediction model for testing, and selecting an optimal prediction model according to a test result; and inputting real-time gas data to predict the gas concentration in real time. According to the parallel prediction method combining the Spark Streaming and the ARIMA-SVM algorithm provided by the invention, the processing of linear and nonlinear data is respectively carried out while the streaming data is rapidly processed, and the prediction timeliness and accuracy are improved.

Description

Technical Field

[0001] The present application relates to a real-time prediction method for gas concentration driven by a Spark Streaming framework, and belongs to the technical field of gas monitoring. Background Art

[0002] In recent years, with the rapid development of information technology and automated control, safety monitoring in coal mining has gradually developed towards intelligence. Traditional monitoring systems can no longer meet the needs of coal mine development, and research on mine gas prediction and early warning has also achieved many results. Gas prediction and early warning are mainly carried out by learning machine learning algorithms.

[0003] The prediction based on learning algorithm needs to rely on training set data, so the training set data needs to have high timeliness. If the data timeliness is insufficient, the accuracy of the prediction result will also be reduced, resulting in a certain lag in data extraction in the research process of gas prediction and early warning. Therefore, how to ensure the timeliness of data extraction and the timeliness of algorithm learning rules in the prediction process is a research direction that urgently needs to be focused on. However, there are relatively few existing studies on the prediction of gas concentration in fully mechanized mining faces. There are many factors that affect the gas concentration in fully mechanized mining faces, and there is a complex nonlinear relationship between the factors. It is one-sided to study the influence of only a single factor on the gas concentration, and multiple influencing factors should be considered comprehensively. In terms of gas concentration prediction, the prediction accuracy of traditional machine learning algorithms such as support vector machines needs to be further improved when facing the problem of gas concentration prediction under the influence of multiple complex factors in fully mechanized mining faces, and it is difficult to deal with a large amount of data. At the same time, in the traditional coal mine gas concentration prediction process, there will be problems such as low timeliness of data and low efficiency of prediction model learning data features, resulting in low accuracy of the final prediction. Summary of the invention

[0004] According to one aspect of the present application, a real-time prediction method for gas concentration driven by a Spark Streaming framework is provided to solve the problems of gas concentration data lag and low efficiency of prediction model learning data features in the gas prediction process of traditional algorithms. The specific solution is:

[0005] A method for real-time prediction of gas concentration driven by a Spark Streaming framework includes the following steps:

[0006] Step 1: Obtain real-time gas concentration data, use the Spark framework to preprocess the obtained gas concentration data to obtain stream data, and divide the stream data into a training set and a test set according to the order in which the gas concentration data is obtained;

[0007] Step 2: Introduce the ARIMA-SVM combination algorithm into the Spark framework to build a gas concentration prediction model, and input the training set obtained in step 1 into the prediction model to train the prediction model;

[0008] Step 3: Input the test set data into the trained gas concentration prediction model, test the gas concentration prediction model, and select the best gas concentration prediction model according to the test results;

[0009] Step 4: Input the acquired real-time gas concentration data into the optimal gas concentration prediction model to make a real-time prediction of the gas concentration.

[0010] Preferably, the construction process of the gas concentration prediction model is:

[0011] The Spark processing framework is built by combining Kafka and Spark Streaming;

[0012] The Spark prediction model is built through Spark MLIib, and the Spark framework is formed by combining the Spark processing framework;

[0013] The ARIMA-SVR combined algorithm is introduced into the prediction model to construct a gas concentration prediction model.

[0014] Preferably, the preprocessing process of real-time gas concentration data is:

[0015] Obtain real-time gas concentration data from gas monitoring sources;

[0016] Use time windows to split the original DStream of the gas monitoring data stream into data RDDs of specified time slices;

[0017] The data RDD serves as the basic unit of the prediction model.

[0018] Preferably, the window length of the data RDD is adjusted according to different data streams to achieve dynamic data update;

[0019] The window length is determined according to the rate and modeling complexity of the streaming data.

[0020] Preferably, the data length and update period of the training set are determined by the length and sliding distance of the Spark Streaming stream window.

[0021] Preferably, in step 2, the construction process of the ARIMA-SVM combination algorithm is:

[0022] Use the ARIMA model to process the historical data of one-dimensional gas time series and obtain the corresponding linear prediction results and residual series;

[0023] The SVM model is used to further analyze and predict the nonlinear factors in the residual series on the panel data that affect the gas time series;

[0024] The analysis and prediction results of the ARIMA model and the SVM model are added together to obtain the final prediction results of the target gas time series data.

[0025] Preferably, the selection of the optimal gas concentration prediction model is achieved through root mean square error.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] This application combines Spark Streaming with the ARIMA-SVM combination algorithm to create a parallel prediction method called the SPARS model. The SPARS model can quickly analyze and predict streaming data, and combines the advantages of the time series model and the support vector machine model in processing linear and nonlinear data, respectively, improving the timeliness of traditional model predictions while also improving accuracy.

[0028] In this application, the Spark Streaming framework is used to process large quantities of real-time streaming data in a short period of time, and the model is continuously updated and optimized so that the model can fully learn the characteristics of the data. At the same time, the advantages of the ARIMA model and the SVM model in processing linear data and nonlinear data are combined, thereby improving the model prediction efficiency while fully reflecting the timeliness of gas prediction. Finally, the proposed new prediction model is verified by the gas data collected on site, and the optimal learning time of the SPARS model in predicting this group of data is determined. The prediction results of the SPARS model are compared and analyzed with the prediction results of ARIMA, SVM and other models to fully confirm that the prediction results have high-precision performance. The introduction of the SPARS model can realize scientific and accurate real-time prediction and analysis of coal mine gas concentration, providing a new idea for coal mines to achieve real-time and accurate prediction of gas. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of the gas concentration prediction framework principle in the embodiment;

[0030] Figure 2 The gas framework structure diagram of Spark Streaming in the embodiment;

[0031] Figure 3-Figure 8 They are respectively the predicted values ​​of gas concentration at 10s, 20s, 30s, 40s, 50s and 60s of the model in the embodiment;

[0032] Fig. 9A comparison chart of gas concentration prediction results at different update cycles in the embodiment;

[0033] Fig.10 This is a graph showing the change in RMSE of the model at different update cycles in the embodiment;

[0034] Fig.11 4 is a comparison chart of prediction results of different models in the embodiments. DETAILED DESCRIPTION

[0035] A method for real-time prediction of gas concentration driven by a Spark Streaming framework includes the following steps:

[0036] Step 1: Obtain gas concentration data (including real-time gas concentration data and historical gas concentration data), and use Spark Streaming in the Spark framework to convert the acquired historical gas concentration data into real-time gas flow data;

[0037] Preprocess the acquired gas concentration data to obtain flow data, and divide the flow data into a training set and a test set according to the order in which the gas concentration data are acquired;

[0038] Step 2: Introduce the ARIMA-SVM combination algorithm into the Spark framework to build a gas concentration prediction model, and input the training set obtained in step 1 into the prediction model to train the prediction model;

[0039] Among them, the construction process of the ARIMA-SVM combination algorithm is:

[0040] Use the ARIMA model to process the historical data of one-dimensional gas time series and obtain the corresponding linear prediction results and residual series;

[0041] The SVM model is used to further analyze and predict the nonlinear factors in the residual series on the panel data that affect the gas time series;

[0042] The analysis and prediction results of the ARIMA model and the SVM model are added together to obtain the final prediction results of the target gas time series data.

[0043] The construction process of the gas concentration prediction model is as follows:

[0044] The Spark processing framework is built by combining Kafka and Spark Streaming;

[0045] The Spark prediction model is built through Spark MLIib, and the Spark framework is formed by combining the Spark processing framework;

[0046] The ARIMA-SVR combined algorithm is introduced into the prediction model to construct a gas concentration prediction model.

[0047] Step 3: Input the test set data into the trained gas concentration prediction model, test the gas concentration prediction model, and select the optimal gas concentration prediction model based on the root mean square error according to the test results;

[0048] Step 4: Input the acquired real-time gas concentration data into the optimal gas concentration prediction model to make a real-time prediction of the gas concentration.

[0049] The ARIMA (p, d, q) (Autoregressive Integrated Moving Average model) model is one of the time series prediction methods. Its main idea is to collect and analyze the observations of past time points, use the model to characterize their internal connections and predict future values. Its prediction of the future can be achieved through the linear equation of its past time values ​​and errors. The construction process of the ARIMA model is:

[0050] Assuming that X = {xi, i = 1, 2, ..., N} is a time series, the ARIMA (p, d, q) model can be described as formula (1):

[0051]

[0052] In formula (1), p, d, and q are required to be non-negative, representing the order of autoregression, the order of difference, and the order of moving average, respectively; x t represents the true value; Represents x t The predicted value of t Represents the predicted error value; represents the parameter value to be estimated for the true value, and θ represents the parameter value to be estimated for the error value. Then ARIMA satisfies formula (2):

[0053]

[0054] In formula (2), Φ(B)=1-φ 1 B-…-φ p B P is the autoregressive coefficient polynomial of the stationary reversible ARIMA(p,q) model; Θ(B)=1-θ 1 B-…-θ 1 Bq is the moving smoothing coefficient polynomial of the stationary reversible ARIM (p, q) model. ARIMA is essentially a combination of difference operation and ARMA model, which has the properties of stationarity and homogeneity of variance.

[0055] The SVM model (Support Vector Machines) is a new machine learning method based on statistical theory. It was proposed by Vapnik in 1995 and was initially used to solve linear separability problems. It was later extended to regression problems. where x i ∈x=R n is the input, y i ∈y=R is the output, then the support vector machine regression model (i.e., SVM model) can be expressed as:

[0056]

[0057] sty-{[w,φ(x i )]-b}≤ε+ξ i Formula (4);

[0058]

[0059] The dual problem is described as formula (7):

[0060]

[0061] In formula (7), The solution is formula (8):

[0062]

[0063] In equations (3) to (8), φ(x i ) represents the mapping function; st represents the condition to be satisfied; ξ i represents the slack variable; ω represents the normal vector; b represents the intercept; C represents the penalty parameter; x i represents the i-th eigenvector; y i represents the class label; α j represents the Lagrange multiplier.

[0064] ARIMA-SVM combination algorithm:

[0065] Since there are both linear and nonlinear trends in the gas concentration time series data, considering that the ARIMA model has its unique advantages in processing linear data and can fully capture the linear part of the time series, SVM has outstanding performance in analyzing and predicting nonlinear data. Therefore, in this application, the ARIMA model is used to process the historical data of the one-dimensional gas time series and obtain the corresponding linear prediction results and residual series; secondly, SVM is used to further analyze and predict the nonlinear factors in the residual series on the panel data that affects the gas time series; finally, the analysis and prediction results of the two models are combined to obtain the final prediction results of the target gas time series data. Given a time series Y = {yk ,k=1,2,…,N} by linear l k and nonlinear nl k Two parts (ie, represented as: y k = l k +nl k ) as attached Figure 1 As shown, the specific combination process is:

[0066] First, the one-dimensional gas data is processed through the ARIMA model to obtain the time series of linear prediction results. and the residual sequence

[0067] Secondly, by further processing the residual time series, a set of time series of nonlinear prediction results is obtained

[0068] Finally, the result of combining linear and nonlinear is the final time series forecast value.

[0069] This application combines the Spark framework with the ARIMA-SVM combination algorithm to form a method for parallel prediction of gas concentration, which is called the SPARS model. The SPARS model can quickly process the gas concentration flow data monitored daily, and combines the advantages of the time series ARIMA model and the support vector machine SVM model to improve the prediction of linear and nonlinear data respectively, improve the timeliness of traditional model predictions, and improve the prediction accuracy through Spark Streaming. This solves the problems of gas concentration data lag and low efficiency of learning data features of the prediction model in the gas prediction process.

[0070] In order to perform predictive modeling on the gas concentration data stream, the distributed stream processing framework Spark Streaming is used in this application to build a real-time prediction system for gas concentration based on the ARIMA-SVM model. Specifically, the real-time data generated by the gas monitoring source is sent to Spark Streaming through the stream generator. In the sliding window calculation provided by Spark Streaming, the gas monitoring data stream is divided into data RDDs of specified time slices using the time window. The DStream in Spark Streaming can perform stream data processing and batch processing at the same time to meet the requirements of various processing types such as data set extraction, machine learning model training and model application. The RDD based on the window length is the basic unit of the prediction model, and the window length can be determined according to the rate of the stream data and the modeling complexity. Thereby, the gas concentration prediction model can be updated as the data stream is dynamically updated.

[0071] The structure of the real-time prediction system for gas concentration based on Spark Streaming is as follows: Figure 2 As shown in the figure, specifically: a distributed mine gas concentration data processing framework (i.e., Spark processing framework) is established in the Spark framework by combining Kafka and Spark Streaming. The purpose of designing the Spark processing framework is to store the gas concentration data monitored in real time during the production process through RDD, and to establish an Executor to pre-process the real-time data by transferring the stream data of gas concentration from the master and slave nodes through the worker process. The processed data is then imported into HDFS and converted into RDD type. A Spark prediction model is constructed through Spark MLIib, and the gas concentration is predicted, error calculated, warned, and data stored by introducing the ARIMA-SVM combined algorithm, thereby realizing real-time prediction and warning of gas concentration in the comprehensive mining face.

[0072] The present application is described in detail below with reference to embodiments, but the present application is not limited to these embodiments.

[0073] Example

[0074] The gas concentration data is obtained through a monitoring source, wherein the monitoring source is a concentration sensor.

[0075] The real-time data generated by the gas monitoring source is sent to Spark Streaming through the stream generator, and the historical gas concentration data is converted into real-time stream data through Spark Streaming. The specific conversion process is as follows:

[0076] In the sliding window calculation provided by Spark Streaming, the gas monitoring data stream is divided into the original DStream into data RDDs of specified time slices using time windows, where RDD is the basic unit of the window length of the prediction model; the window length can be determined according to the rate of the stream data and the modeling complexity; the structure of the real-time prediction system of gas concentration based on Spark Streaming is as follows: Figure 4 As shown in the figure, the system HDFS is used for storage, and it can also be sent directly to Spark Streaming. Coal mine gas and related sensors transmit real-time monitoring data to the system through the network. The data stream can be written into the distributed storage and combined with Spark MLIib to establish a prediction model. The model can be dynamically updated with the data stream, and finally the ARIMA-SVR combined algorithm is used to perform real-time prediction of gas concentration.

[0077] In this embodiment, the data set collected in the first 3 hours is used as the training data set of the model for estimating the model parameters, and the data set in the last hour is used as the test set of the model for checking the fit of the prediction model.

[0078] In order to simulate the dynamic gas data flow monitored by coal mine gas sensors, the gas concentration sequence data of the training set is sent to Spark Streaming through TCP socket at a rate of 0.2 / s, thereby simulating the real-time dynamic data flow of gas concentration; Spark Streaming uses the built-in data stream source socketTextStream to receive the gas concentration stream data as the input source.

[0079] When predicting gas concentration, the length and sliding distance of the Spark Streaming window can be specified according to the requirements of the actual application to determine the training data length and update period of the stream regression model.

[0080] In order to verify the real-time and accuracy of the flow regression model, this embodiment predicts and analyzes the gas concentration data by setting different model update cycles. The flow regression window length used in the experiment is 1 minute, and the model update cycle is set to 10s, 20s, 30s, 40s, 50s, and 60s respectively to perform real-time prediction of gas concentration.

[0081] The experiment was conducted six times to simulate the prediction accuracy of gas concentration data under different model update cycles. The prediction results are as follows: Figure 3-Figure 8 As shown in the figure, real represents the real value of gas concentration, and predict represents the predicted value. It can be concluded that the degree to which the mean of the model prediction value is close to the mean of the real value at different update times is 60s, 50s, 40s, 30s, 20s, and 10s respectively;

[0082] Fig. 9 Comparison of gas prediction results for different model update cycles. Fig. 9 It can be seen that when the update time is 50s and 60s, the degree of fit between the predicted value and the true value is high. That is, from a general point of view, the longer the model update time, the larger the training set data flow, the more fully the model training, and therefore the higher the prediction accuracy; on the contrary, the shorter the model update time, the smaller the training set data flow, the insufficient model training process, resulting in low prediction accuracy. It can be concluded that as the model update cycle expands, the model's ability to predict gas is gradually enhanced, the degree of fit between the predicted value and the true value is enhanced, and the predicted value is more accurate. However, if the model update time is infinitely extended, it cannot meet the requirements of real-time data flow processing. The principle of real-time data flow processing should be that when the model update time is relatively short, the prediction accuracy is relatively high. Therefore, the above results need to be further discussed.

[0083] In order to further determine the corresponding relationship between the model update cycle and the prediction results, the root mean square error (RMSE) is used to evaluate the model. The precision RMSE of the statistical model is the square root of the mean of the sum of squared errors. The calculation formula is shown in formula (9):

[0084]

[0085] In formula (9), n represents the number of samples; y i represents the true value; Represents the predicted value.

[0086] In the prediction process of the test set above, when the model update time is 60s, the model is updated 60 times. Similarly, when the model update time is 50s, 40s, 30s, 20s, and 10s, the model is updated 72 times, 90 times, 120 times, 180 times, and 360 times, respectively. The RMSE of each updated model is calculated. The changes in the RMSE of the model under different update cycles are as follows Fig.10 As shown, Fig.10 The lower horizontal axis represents the number of model updates, the vertical axis is divided into six groups, all representing RMSE values, and the upper horizontal axis represents the update time of the six groups of models. The lower horizontal axis and the vertical axis constitute the trend of the change of the RMSE value of the model under different update cycles, and the upper horizontal axis and the vertical axis constitute the trend chart of the change of the overall RMSE value of each model in the 1h prediction process. The overall value of the RMSE of the model under different update cycles is recorded in Table 1:

[0087] Table 1 Overall values ​​of RMSE of the model under different update cycles

[0088]

[0089]

[0090] It can be seen from Table 1 that when the model update time is 50s, its value is the smallest 0.01211, and when the update time is 10s, its value is the largest 0.02617. It can be concluded that in the overall prediction process, when the model update time is 50s, the prediction accuracy is higher.

[0091] In order to further explore the superiority of the SPARS model prediction, the following comparative examples were verified.

[0092] The data in the above embodiment 1 are predicted by a simple ARIMA model, an SVM model, and an ARIMA-SVM combined model respectively:

[0093] Similarly, the gas data of the first 3 hours is set as the training set, and the gas data of the last hour is set as the test set. The prediction results are compared and analyzed with the real value data sequence and the prediction results of the SPARS model. The prediction results are as follows: Fig.11 shown.

[0094] Depend on Fig.11 It can be seen that the prediction effects of the ARIMA model and the SVM model are relatively poor, because the ARIMA model can show good fit for the linear data part, but the prediction results for the nonlinear part of the data are poor; the SVM model shows good fit for the nonlinear part of the data, but the prediction results for the linear part are relatively poor; the ARIMA-SVM combination model shows relatively good prediction results, because the ARIMA-SVM combination model integrates the advantages of linear data and nonlinear data prediction in the above-mentioned ARIMA model and SVM model, and therefore shows a good prediction trend; and the prediction trend of the SPARS model described in this application is closest to the true value sequence, because the model is updated every 50 seconds according to the time series characteristics, and can more accurately capture the characteristics of the time series of each time period than the non-real-time prediction model, thereby achieving accurate prediction.

[0095] The prediction accuracy of the above four prediction models is quantitatively analyzed, and the maximum error, minimum error and average error of the four prediction models are obtained, as shown in Table 2.

[0096] Table 2 Comparison of prediction results errors of different models

[0097]

[0098]

[0099] It can be concluded from Table 2 that the error value of the SPARS model constructed in this application is the smallest, so the prediction accuracy of this model is the highest.

[0100] The above are only a few embodiments of the present application and do not constitute any form of limitation to the present application. Although the present application is disclosed as above with preferred embodiments, it is not intended to limit the present application. Any technician familiar with the profession, without departing from the scope of the technical solution of the present application, using the technical content disclosed above to make slight changes or modifications are equivalent to equivalent implementation cases and fall within the scope of the technical solution.

Claims

1. A real-time prediction method for gas concentration driven by Spark Streaming framework, characterized in that: The following steps are involved: Step 1: Obtain gas concentration data, use the Spark framework to preprocess the obtained gas concentration data to obtain stream data, and divide the stream data into a training set and a test set according to the order in which the gas concentration data is obtained; Step 2: Introduce the ARIMA-SVM combination algorithm into the Spark framework to build a gas concentration prediction model, and input the training set obtained in step 1 into the prediction model to train the prediction model; Step 3: Input the test set data into the trained gas concentration prediction model, test the gas concentration prediction model, and select the best gas concentration prediction model according to the test results; Step 4: Input the acquired real-time gas concentration data into the optimal gas concentration prediction model to make a real-time prediction of the gas concentration.

2. According to the method for real-time prediction of gas concentration driven by Spark Streaming framework in claim 1, it is characterized in that: The construction process of the gas concentration prediction model is as follows: The Spark processing framework is built by combining Kafka and Spark Streaming; The Spark prediction model is built through Spark MLIib, and the Spark framework is formed by combining the Spark processing framework; The ARIMA-SVR combined algorithm is introduced into the prediction model to construct a gas concentration prediction model.

3. According to claim 2, a method for real-time prediction of gas concentration driven by a Spark Streaming framework is characterized in that: The preprocessing process of real-time gas concentration data is as follows: Obtain real-time gas concentration data from gas monitoring sources; Use time windows to split the original DStream of the gas monitoring data stream into data RDDs of specified time slices; The data RDD serves as the basic unit of the prediction model.

4. According to claim 3, a method for real-time prediction of gas concentration driven by a Spark Streaming framework is characterized in that: The window length of the data RDD is adjusted according to different data streams to achieve dynamic data update; The window length is determined according to the rate and modeling complexity of the streaming data.

5. According to claim 3, the method for real-time prediction of gas concentration driven by Spark Streaming framework is characterized in that: The data length and update period of the training set are determined by the length and sliding distance of the Spark Streaming stream window.

6. According to the method for real-time prediction of gas concentration driven by Spark Streaming framework in claim 1, it is characterized in that: In step 2, the construction process of the ARIMA-SVM combination algorithm is: Use the ARIMA model to process the historical data of one-dimensional gas time series and obtain the corresponding linear prediction results and residual series; The SVM model is used to further analyze and predict the nonlinear factors in the residual series on the panel data that affect the gas time series; The analysis and prediction results of the ARIMA model and the SVM model are added together to obtain the final prediction results of the target gas time series data.

7. According to claim 1, a method for real-time prediction of gas concentration driven by a Spark Streaming framework is characterized in that: The selection of the optimal gas concentration prediction model is achieved through the root mean square error.

Images