Landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization

Abstract

The present invention belongs to the technical field of disaster prevention and reduction, and discloses a landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization, including: obtaining original monitoring data from the monitoring area and preprocessing, obtaining monitoring data, converting monitoring data of different data structures of different databases into target data in a preset format, building a Flink cluster to integrate target data from various databases into a target database in the form of stream computing, collecting system information based on an improved PSO task scheduling optimization algorithm, counting resource performance data of each node of the Flink cluster, obtaining load data of each node, building an LSTM model, predicting and analyzing the load of the Flink cluster, and optimizing task scheduling of the Flink cluster using a particle swarm optimization algorithm according to the results of the load analysis, and performing comparative analysis before and after optimization. The present invention realizes real-time integration of multi-source heterogeneous disaster monitoring data.

Description

Technical Field

[0001] The present invention belongs to the technical field of disaster prevention and mitigation, and specifically relates to a landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization. Background Art

[0002] my country has a vast territory, complex geographical and climatic conditions, and is strongly affected by the monsoon climate. Landslide refers to the loosening of the rock or soil at the slope under the action of gravity due to the influence of groundwater activity, rainfall, earthquakes and other factors, resulting in overall or scattered sliding. Long-term monitoring in landslide-prone areas and the establishment of an early warning system can predict the possibility of landslides and thus reduce the occurrence of landslide disasters. During the monitoring process, a large amount of static or dynamic data is generated. This type of data has a variety of data types, complex data sources, and large differences in data formats. It is usually scattered in different types of databases, so that these data need to go through a series of complex and tedious processing processes to meet people's requirements for system operations. Due to the lack of data integration standards and data service methods, these data that consume a lot of human and material resources cannot be shared, and the problem of integrating multi-source heterogeneous data of landslide disasters needs to be solved urgently.

[0003] With the continuous development of big data technology, various real-time data processing frameworks have emerged, such as Storm, Spark, and Flink. At present, although these frameworks can solve the problem of streaming big data processing very well, their default task scheduling mechanism cannot fully consider various factors such as their own performance and job structure in actual use, and does not bring out the maximum performance of the cluster. Therefore, optimizing the task scheduling strategy to effectively improve data computing and transmission performance, improve execution efficiency, and reasonably allocate tasks to resource execution are issues that need to be focused on at present.

[0004] Based on this, the present invention provides a landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization. Summary of the invention

[0005] In response to the above-mentioned technical problems, the present invention provides a landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization, aiming to optimize task scheduling through an improved PSO algorithm, so as to give full play to the maximum performance of the cluster to perform real-time / quasi-real-time integration of multi-source heterogeneous monitoring data of landslide disasters, and provide more effective guidance for disaster prevention and mitigation work.

[0006] The invention provides a landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization, the method comprising the following steps:

[0007] Step 1: Obtain original monitoring data from each monitoring device at each monitoring point in the monitoring area and store them in a corresponding database;

[0008] Step 2: Perform a first preprocessing on each type of original monitoring data to obtain monitoring data;

[0009] Step 3: converting the monitoring data with different data structures in different databases into target data in a preset format;

[0010] Step 4: Build a Flink cluster and integrate all target data from various databases into the target database in the form of stream computing;

[0011] Step 5: Based on the improved PSO task scheduling optimization algorithm, collect the system information of each node in the Flink cluster, count the resource performance data of each node, obtain the load data of each node and the average load of the Flink cluster;

[0012] Step 6: Build an LSTM model to predict and analyze the load of the Flink cluster.

[0013] Step 7: According to the results of the load analysis, the particle swarm optimization algorithm is used to optimize the task scheduling of the Flink cluster, and a comparative analysis is performed before and after the optimization.

[0014] Specifically, the monitoring equipment includes at least a rain gauge, a piezometer, a mud level meter, a multi-point displacement meter and GPS monitoring.

[0015] Specifically, in step 2, the first preprocessing at least includes cubic spline interpolation processing, least squares method and time series interpolation.

[0016] Specifically, the step 3 includes:

[0017] Step 31, respectively obtaining the monitoring data of different data structures in different databases through a data access component;

[0018] Step 32, uniformly convert the data format through format conversion, mapping conversion and / or conversion conversion methods to obtain the target data in the preset format, wherein format conversion is used for the situation where the data structure of the monitoring data and the data structure of the corresponding target data are different; mapping conversion is used for the situation where the fields of the data table of the monitoring data and the data table of the corresponding target data are different; conversion conversion is used for the situation where the monitoring data is data obtained by direct monitoring and needs to be converted by calculation.

[0019] Specifically, the Flink cluster consists of 4 machines, including 1 master node and 3 slave nodes, wherein the master node is the JobManager node in the Flink cluster, and the 3 slave nodes are the TaskManager nodes in the Flink cluster. Each node is connected through Gigabit Ethernet, and the operation mode between the nodes is Yarn mode.

[0020] Specifically, the resource performance data includes CPU utilization and memory usage.

[0021] Specifically, the calculation formula for the node load data is:

[0022] L j =θ c L cj +θ m L mj

[0023] Among them, L j is the load data of the jth node, θ c and θ m is the parameter weight coefficient, θ c +θ m =1,L cj is the CPU utilization, L mj is the memory usage, j is a positive integer from 1 to J, and J is the total number of nodes in the Flink cluster;

[0024] The calculation formula for the average load of the Flink cluster is:

[0025]

[0026] Where L is the average load.

[0027] Specifically, the step 6 includes:

[0028] Step 61: Collect a cluster load time series data set with a preset time interval and a length of N, and perform a second preprocessing on the cluster load time series data set;

[0029] Step 62: Divide the preprocessed data set into a training set and a test set;

[0030] Step 63: Use the deep learning framework TensorFlow to build an LSTM model, determine the parameters that need to be optimized, and determine the optimization range of each parameter, wherein the parameters that need to be optimized include the number of neurons, the learning rate, and the number of training iterations;

[0031] Step 64: Use the training set to train the LSTM model;

[0032] Step 65: Use the test set to evaluate the trained LSTM model;

[0033] Step 66: Input the load data of each node into the trained LSTM model to obtain the load prediction result within a preset time.

[0034] Specifically, the second preprocessing includes removing outliers and / or filling missing values.

[0035] Specifically, in the particle swarm optimization algorithm, a linear decreasing inertia weight is used.

[0036] Compared with the prior art, the beneficial effects of the present invention are at least as follows:

[0037] 1. Based on the Flink streaming processing framework, the real-time collection and acquisition of multi-source heterogeneous landslide monitoring data is realized, and the data is processed, distributed and shared efficiently and reliably, ensuring low latency and efficient data processing.

[0038] 2. The abnormal data are processed by cubic spline interpolation, least squares method and time series interpolation methods to ensure the authenticity of landslide deformation monitoring data, which is of great significance for successfully warning landslides and improving the accuracy of landslide warnings.

[0039] 3. The monitoring data with different data structures obtained from different databases are converted into a unified format through format conversion, conversion and other methods, which greatly facilitates the unified analysis and management of monitoring data and effectively solves the problem of unified management of multi-source heterogeneous monitoring data.

[0040] 4. The default task scheduling mechanism of Flink is optimized through the improved PSO algorithm. While realizing the real-time integration of multi-source heterogeneous monitoring data, tasks are reasonably allocated to resource execution, further improving the data processing efficiency and cluster execution efficiency, and bringing into play the maximum performance of the cluster, thereby enhancing the ability to respond to landslide disasters and risk management. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0042] Figure 1 It is a technical architecture diagram of the method of the present invention;

[0043] Figure 2 is a flow chart of the method of the present invention;

[0044] Figure 3 It is a flow chart of the cluster load balancing strategy based on improved PSO of the present invention;

[0045] Figure 4 It is the inertia weight variation curve diagram of the improved PSO algorithm of the present invention;

[0046] Figure 5 It is a comparison diagram before and after the missing data processing of the present invention;

[0047] Figure 6 It is a comparison diagram before and after the data denoising process of the present invention;

[0048] Figure 7 It is a comparison chart of LSTM load prediction results of the present invention;

[0049] Figure 8 It is a comparison chart of execution time of several task scheduling algorithms of the present invention;

[0050] Fig. 9 This is a throughput comparison chart of several task scheduling algorithms of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the specific embodiments described herein are only used to explain the present invention and are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.

[0052] It should be noted that if there are descriptions involving "first", "second", etc. in the embodiments of the present invention, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0053] Figure 1 Shown is a technical architecture diagram of a landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization provided by the present invention.

[0054] Figure 2FIG. 1 is a flow chart of an embodiment of a landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization provided by the present invention, and the flow chart specifically includes the following steps:

[0055] Step 1: Obtain original monitoring data from each monitoring device at each monitoring point in the monitoring area and store them in the corresponding database.

[0056] Specifically, the monitoring equipment includes at least a rain gauge, a piezometer, a mud level meter, a multi-point displacement meter and GPS monitoring.

[0057] Data is collected through Flink and Kafka message queues to obtain multi-source, heterogeneous landslide disaster monitoring data in the study area in real time, including a variety of monitoring instruments such as rain gauges, piezometers, mud level meters, multi-point displacement meters, GPS monitoring, etc., and pre-processing is performed to facilitate subsequent integration work.

[0058] The original monitoring data collected by each monitoring instrument is stored in the database corresponding to the monitoring instrument.

[0059] Step 2: Perform a first preprocessing on each type of original monitoring data to obtain monitoring data.

[0060] Specifically, in step 2, the first preprocessing at least includes cubic spline interpolation processing, least squares method and time series interpolation.

[0061] The abnormal data processing module identifies and processes the acquired multi-source heterogeneous data through data smoothing, abnormal elimination and other methods. This module integrates several common abnormal data processing methods, including interpolation processing, equal spacing (equal time interval) processing, data denoising processing (accumulation generation method), etc.

[0062] Among them, cubic spline interpolation is used for missing data processing. The missing monitoring values ​​in the original monitoring data due to interference from external factors are interpolated. For n+1 data points (x0, y0), (x1, y1), ..., (x n ,y n ), cubic spline interpolation will be the entire interpolation interval [x0, x n ] is divided into multiple small intervals, and a cubic polynomial function is used to fit the data points in each small interval, so there are n cubic polynomial functions in total. i , x i+1 ], using a cubic polynomial function to fit the data points:

[0063] S(x)=S i (x) = a i +b i (xx i )+ci (xx i ) 2 +d i (xx i ) 3

[0064] Then substitute the point x corresponding to the missing value into the above formula to get the approximate value of the missing value.

[0065] The least squares method is used for data denoising. By minimizing the sum of squares of the errors to find the best function match for the data, the monitoring values ​​that have undergone sudden changes due to interference from external factors in the original monitoring data are denoised, thereby reflecting the true situation of landslide deformation. Assume that there is an approximate curve f(x) = a n x n +a n-1 x n-1 +…+a1x+a0, so that f best fits each observation point, then the vertical distance between a point P1(x1, y1) on the plane and the curve is D1=|f(x1)-y1|, then the sum of the squares of the vertical distances from all points to the line is:

[0066] D 2 =(f(x1)-y1) 2 +(f(x2)-y2) 2 +…+(f(x n )-y n ) 2

[0067] Determine the appropriate n} makes D 2 Has a minimum value.

[0068] Time series interpolation is used for equal interval processing of time series. The non-equal interval time series caused by the failure to detect the data of a certain day in the original monitoring data due to various reasons are interpolated using the interpolation method.

[0069] Step 3: Convert the monitoring data with different data structures in different databases into target data in a preset format.

[0070] The source of monitoring data can be data from different databases or data obtained from different monitoring instruments. The data in different formats obtained from different monitoring instruments are converted to obtain data in the same format, which solves the problem that the data obtained from different instruments cannot be used uniformly due to incompatibility in landslide disaster monitoring, and realizes the rapid, real-time and effective integration of these multi-source heterogeneous monitoring data.

[0071] Specifically, the step 3 includes:

[0072] Step 31: Obtain the monitoring data of different data structures in different databases through the data access component.

[0073] Step 32, uniformly convert the data format through format conversion, mapping conversion and / or conversion conversion methods to obtain the target data in the preset format, wherein format conversion is used for the situation where the data structure of the monitoring data and the data structure of the corresponding target data are different; mapping conversion is used for the situation where the fields of the data table of the monitoring data and the data table of the corresponding target data are different; conversion conversion is used for the situation where the monitoring data is data obtained by direct monitoring and needs to be converted by calculation.

[0074] The heterogeneous data processing module uses the same data standard to convert and convert the monitoring data of different databases with different data structures, and then collects them into a unified database. First, through the data access component (to access multiple types of databases at the same time, that is, the function of multi-database support, so as to obtain monitoring data with different data structures in different databases); then the data format is uniformly converted through format conversion, mapping conversion and conversion conversion.

[0075] The monitoring data comes from different monitoring instruments, including xlsx (xls), csv, txt, doc (docx), etc. For files in these formats, corresponding mapping relationships are established through format conversion to unify data standards.

[0076] If the data stored in the source data table are original records sent back by monitoring instruments, it is necessary to extract the corresponding formulas from the database and the parameters from the parameter table according to different monitoring instruments, convert the original records into physical quantities, and then perform the warehousing operation. This is the most common situation.

[0077] Different processing methods for different raw data help to obtain target data in the same format more quickly.

[0078] Step 4: Build a Flink cluster and integrate all target data from various databases into the target database in the form of stream computing.

[0079] Specifically, the Flink cluster consists of 4 machines, including 1 master node and 3 slave nodes, wherein the master node is the JobManager node in the Flink cluster, and the 3 slave nodes are the TaskManager nodes in the Flink cluster. Each node is connected through Gigabit Ethernet, and the operation mode between the nodes is Yarn mode.

[0080] Step 5: Based on the improved PSO task scheduling optimization algorithm, collect the system information of each node in the Flink cluster, count the resource performance data of each node, and obtain the load data of each node and the average load of the Flink cluster.

[0081] Specifically, the resource performance data includes CPU utilization and memory usage.

[0082] Specifically, the calculation formula of the node load data is:

[0083] L j =θ c L cj +θ m L mj

[0084] Among them, L j is the load data of the jth node, θ c and θ m is the parameter weight coefficient, θ c +θ m =1,L cj is the CPU utilization, L mj is the memory usage, j is a positive integer from 1 to J, and J is the total number of nodes in the Flink cluster;

[0085] The calculation formula for the average load of the Flink cluster is:

[0086]

[0087] Where L is the average load.

[0088] θ c and θ m Weigh the load impact of different processing tasks. Different tasks have different requirements for memory and CPU during processing. The weight coefficients of memory and CPU load impact of different processing tasks are different. According to the actual situation, θ c and θ m Adjustments are made to reflect the impact of both CPU and memory on node load.

[0089] The load of a single node j in the Flink cluster can be expressed as L j = <L cj , L mj >.

[0090] Perform cluster resource monitoring. Cluster resource monitoring collects important system information for the task scheduling optimization algorithm based on the improved PSO, and collects statistics on the resource performance data of each node in the cluster, including CPU utilization, memory usage, task queue length, etc., which reflects the current load capacity of the node. These resource performance data will be used as input for load prediction and strategy optimization.

[0091] Step 6: Build an LSTM model to predict and analyze the load of the Flink cluster.

[0092] Long short-term memory (LSTM) is a special recurrent neural network (RNN) that has the ability to memorize long-term and short-term information. Compared with traditional RNN, LSTM has stronger memory ability and can capture and process long-term dependencies. LSTM implements the memory and forgetting mechanism by introducing a "gate" structure. Each gate consists of an activation function and an element-by-element multiplication operation to control the flow of information.

[0093] The main components of LSTM include:

[0094] (1) Input gate i t (Input Gate): Determines what information will be input into the cell state.

[0095] (2) Forget gate f t (Forget Gate): Determines which information will be forgotten or discarded.

[0096] (3) Cell state C t (Cell State): used to store and transmit information.

[0097] (4) Output gate O t (Output Gate): Determines which parts of the cell state will be output.

[0098] At time t, the calculation method of the LSTM model is:

[0099]

[0100] Among them, W i , W f , W o , W c are the weight matrices of the input gate, forget gate, output gate and candidate memory, respectively, h t-1 is the hidden state of the previous time step, h t is the hidden state at the current time step, which is controlled by the output gate and the tanh transform of the memory, xt is the input feature of the current time step, b i 、b f 、b o 、b c are the bias vectors of the input gate, forget gate, output gate and candidate memory, respectively, C t is the candidate memory of the current step, uses the tanh activation function to transform the input, and o is the sigmoid activation function.

[0101] The core idea of ​​the LSTM model is to control the flow of information and the updating of memory through a gating mechanism, so as to better handle long-term dependencies, which makes LSTM perform well in processing time series data.

[0102] Specifically, the step 6 includes:

[0103] Step 61 : Collect a cluster load time series data set with a preset time interval and a length of N, and perform a second preprocessing on the cluster load time series data set.

[0104] Step 62: Divide the preprocessed data set into a training set and a test set.

[0105] Step 63: Use the deep learning framework TensorFlow to build an LSTM model, determine the parameters that need to be optimized, and determine the optimization range of each parameter, wherein the parameters that need to be optimized include the number of neurons, the learning rate, and the number of training iterations.

[0106] Step 64: Use the training set to train the LSTM model.

[0107] Step 65: Use the test set to evaluate the trained LSTM model.

[0108] Step 66: Input the load data of each node into the trained LSTM model to obtain the load prediction result within a preset time.

[0109] The preset time interval and the preset time are set according to the experience of those skilled in the art or according to the actual application scenario, and the embodiments of the present application do not limit this. For example, the preset time interval is 10s.

[0110] Preferably, the cluster load time series data set is a cluster average load time series data set. The length N of the cluster load time series data set is related to the total acquisition time. For example, if the total acquisition time is 5 minutes and the preset time interval is 5 seconds, the length N is 60.

[0111] Using the training set F trainThe LSTM model is trained. During the training process, the load data is used as the input sequence, and the load situation of the next time step is predicted as output. The mean square error (MSE) is used as the loss function for model training. The calculation formula of the mean square error MSE is as follows:

[0112]

[0113] Among them, n represents the number of samples in the training set involved in model training. Represents the predicted value of each sample load, y i Represents the true value of each sample load.

[0114] Use the test set F test The trained LSTM model is evaluated by calculating the root mean square error (RMSE) between the predicted value and the true value to evaluate the prediction performance. The calculation formula of the root mean square error RMSE is as follows:

[0115]

[0116] Among them, n represents the number of samples in the test set involved in model training. Represents the predicted value of each sample load, y i Represents the true value of each sample load.

[0117] Use the trained LSTM model for load prediction. In actual applications, input the latest load data collected in real time into the model to obtain the load prediction results for a period of time in the future, which facilitates cluster load balancing decisions.

[0118] Specifically, the second preprocessing includes removing outliers and / or filling missing values.

[0119] Step 7: According to the results of the load analysis, the particle swarm optimization algorithm is used to optimize the task scheduling of the Flink cluster, and a comparative analysis is performed before and after the optimization.

[0120] Specifically, in the particle swarm optimization algorithm, a linear decreasing inertia weight is used.

[0121] The flow chart of cluster load balancing strategy based on improved PSO is as follows Figure 3 shown.

[0122] The inertia weight change curve of the improved PSO algorithm is as follows: Figure 4 shown.

[0123] Flink's default task scheduling algorithm is the first-come, first-served algorithm, which ignores the relationship between task resource requirements and node available resources, resulting in uneven task loads on different nodes, thus affecting system throughput.

[0124] Firstly, the load of machines in the cluster is analyzed. For machines with overload in the cluster, the load balancing of the cluster is achieved through the improved PSO-based method.

[0125] For the basic particle swarm algorithm, the solution space is divided into a group of particles, each particle represents a candidate solution. In a d-dimensional solution space, n particles are initialized, and the set of candidate solutions can be expressed as X = (X1, X2, ..., X n ). The position of each particle represents the position of the candidate solution in the solution space, where the position of the i-th particle can be expressed as X i =(X i1 , X i2 , …, X in ). The velocity of each particle represents the moving direction and speed of the particle in the solution space, where the velocity of the i-th particle can be expressed as V i =(V i1 , V i2 , …, V in ). Randomly initialize the position and velocity of the particles, and set the initial optimal position of each particle to the current position, denoted by P i =(P i1 , F i2 , …, F in ). For each particle, the fitness value is calculated according to its position, which measures the quality of the candidate solution. Record the global optimal historical position, denoted as P g =(P g1 , P g2 ,…,P gn ), which can be set to any particle position at the beginning, or the position corresponding to the best fitness value in the group. For each particle i, the speed and position of the particle are updated according to the current speed, current position, individual historical optimal position and global historical optimal position, as shown below:

[0126] V i，t+1 =ω*V i，t +c1*r1*(F i -X i，t )+c2*r2*(P g -X i，t )

[0127] X i，t+1 =X i，t +V i，t+1

[0128] Where i = 1, 2, ..., n; V i，t is the velocity of particle i at time t; X i，tis the position of particle i at time t; ω is the inertia weight, which is used to balance the historical speed and current speed of the particle; c1 and c2 are acceleration factors, which control the degree to which the particle approaches the individual optimal solution and the group optimal solution; r1 and r2 are random numbers distributed between [0, 1], which are used to increase randomness and diversity.

[0129] Update the historical optimal position of individuals and groups, if f(X i，t+1 )>f(P i ), then update P i =X i，t+1 ; If f(X i，t+1 )>f(P g ), then update P g =X i，t+1 , repeat the above steps until the maximum number of iterations is reached or the target fitness value is reached, and the optimal solution in the search space is obtained.

[0130] In the particle swarm algorithm, the inertia weight ω is used to balance the exploration and utilization of the solution space by the particle's historical speed and current speed. The choice of inertia weight has a certain influence on the convergence and search function of the algorithm. The larger the ω coefficient, the stronger the global optimization ability, and the smaller the ω coefficient, the stronger the local optimization ability. Therefore, a fixed ω coefficient weakens the algorithm's global optimization ability and slows down the algorithm's convergence speed.

[0131] The present invention uses linear decreasing inertia weight to improve the performance of PSO:

[0132]

[0133] Among them, ω max and ω min are the maximum and minimum values ​​of ω respectively, t is the current generation number, t max is the maximum number of iterations.

[0134] The research area selected for the present invention is the reservoir area of ​​Dagangshan Hydropower Station on the Dadu River in Sichuan Province, which is located in Shimian County, Ya'an City, Sichuan Province in the upper reaches of the middle reaches of the Dadu River. It is the 14th cascade power station planned for the mainstream of the Dadu River. The data used in the present invention include GNSS monitoring of five deformation bodies of Zhengjiaping, Huangcaoping, Xinhua, Mogangling and Dagou, rain gauges, piezometers, mud level meters, multi-point displacement meters and other multi-source heterogeneous monitoring data.

[0135] Figure 5(a)-(d) and (e)-(h) show the comparison of missing data before and after processing in the GNSS automated monitoring data of Huangcaoping and Zhengjiaping deformable bodies in 2016. The cubic spline interpolation method and Lagrange interpolation method are used to interpolate the missing data. By comparing with the Lagrange interpolation method, it can be seen that the data after the cubic spline interpolation method maintains the same trend as the original data, while the data after the Lagrange interpolation method is quite different from the original data. Among them, the average MSE of the Lagrange interpolation method in (a)-(h) is 9.586, while the average MSE of the cubic spline interpolation method is 3.425. Therefore, the result of using the cubic spline interpolation is closer to the original data and has a better fit.

[0136] Figure 6 (a)-(d) and (e)-(g) respectively show the comparison of the "noise" data before and after processing in the GNSS automated monitoring data of Mogangling and Xinhua deformation in 2016. The original data were fitted using the least squares method and the moving average method to remove the noise in the monitoring data. It can be seen from the figure that the data after denoising using the moving average method still has a certain degree of jumpiness, while the least squares method can better remove the noise in the data and is more in line with the original trend of the data.

[0137] Figure 7 This is the LSTM load prediction result analysis diagram. By analyzing the load data, the CPU component is set to affect θ c and memory components affect θ m The load data of the cluster during the 40-minute operation is used as the time series data set of the cluster load prediction model, and the first 80% of the data is selected as the training set F train , the last 20% of the data is used as the test set F test After preprocessing the acquired cluster historical load data, the LSTM model and BP model are used to predict the cluster load. As can be seen from the figure, the prediction results of the BP model and the LSTM model are basically consistent with the trend of the original sequence, but the prediction accuracy of the BP model is obviously not as good as that of the LSTM model. The error between the predicted value and the true predicted value obtained by the LSTM model after training is mostly concentrated between -5.0 and 5.0, and the peak prediction is basically accurate with a small error. Therefore, it is verified that the effect of the LSTM model in predicting the cluster load is in line with expectations.

[0138] Figure 8This is a comparison chart of the execution time of several task scheduling algorithms. The execution time refers to the total time required from task submission to completion. The default polling scheduling algorithm of Flink, the PSO algorithm, the improved PSO algorithm and the GA algorithm are compared in terms of the overall task execution time. The experimental data is divided into different data sizes according to time, namely Data1, Data2, and Data3. The data sizes are 2GB, 5GB, and 8GB respectively, and the degree of parallelism is set to 8. Figure 7 The experimental results show that under different data scales, the execution time of the basic PSO algorithm and the GA algorithm is not much different, but both algorithms are more efficient than the default polling scheduling algorithm of Flink, which is about 2.8% higher; and the improved PSO algorithm is shorter than the default polling scheduling algorithm of Flink, the basic PSO algorithm and the GA algorithm. In the test of multiple data sets, the average optimization efficiency of task execution time when the improved PSO algorithm is used is finally found to be 4.7%.

[0139] Fig. 9 This is a throughput comparison chart of several task scheduling algorithms. Throughput refers to the amount of data that the system can process per unit time, which represents the load capacity of the system. During the experiment, different degrees of parallelism were set to compare the throughput of Flink's default polling scheduling algorithm, PSO algorithm, improved PSO algorithm and GA algorithm at different degrees of parallelism. Figure 8 It can be seen that when the parallelism is 8 and 12, the throughput of the GA algorithm and the Flink default algorithm is not much different. When the parallelism is 18, the parallelism of the GA algorithm is slightly lower than that of the Flink default algorithm. Under the three degrees of parallelism, the throughput of the basic PSO algorithm is better than that of the Flink default algorithm. After testing on various data sets, the average optimization rate is 3.2%. However, the throughput of the improved PSO algorithm is higher than that of the Flink default algorithm, the basic PSO algorithm and the GA algorithm. In comparison, the optimization effect of the improved PSO algorithm is the best. After testing on various data sets, the average optimization efficiency of the throughput of the improved PSO algorithm is 5.4%.

[0140] The above results prove that the real-time / quasi-real-time integration method of landslide disaster monitoring data based on Flink of the present invention integrates landslide disaster monitoring data. In view of the problem that the default first-come-first-served task scheduling mechanism of Flink leads to uneven loads on different nodes in the cluster, thus affecting the system throughput, a load balancing strategy based on improved particle swarm is proposed to optimize the default task scheduling mechanism of Flink. The experimental results show that compared with the default scheduling algorithm of Flink, the strategy proposed by the present invention greatly improves the resource utilization and task execution efficiency of the system in the process of landslide disaster monitoring data integration.

[0141] It should be understood that, although each step in the flow chart of each embodiment of the present invention is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0142] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The above-mentioned program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0143] The above-mentioned embodiments only express the preferred implementation mode of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A landslide disaster multi-source monitoring data integration method based on Flink and task scheduling optimization, characterized in that: The steps include: Step 1: Obtain original monitoring data from each monitoring device at each monitoring point in the monitoring area and store them in a corresponding database; Step 2: Perform a first preprocessing on each type of original monitoring data to obtain monitoring data; Step 3: converting the monitoring data with different data structures in different databases into target data in a preset format; Step 4: Build a Flink cluster and integrate all target data from various databases into the target database in the form of stream computing; Step 5: Based on the improved PSO task scheduling optimization algorithm, the system information of each node in the Flink cluster is collected, the resource performance data of each node is counted, and the load data of each node and the average load of the Flink cluster are obtained; Step 6: Build an LSTM model to predict and analyze the load of the Flink cluster. Step 7: According to the results of the load analysis, the particle swarm optimization algorithm is used to optimize the task scheduling of the Flink cluster, and a comparative analysis is performed before and after the optimization.

2. The method according to claim 1, characterized in that The monitoring equipment at least includes a rain gauge, a piezometer, a mud level meter, a multi-point displacement meter and GPS monitoring.

3. The method according to claim 1, characterized in that In the step 2, the first preprocessing at least includes Lagrange interpolation processing, least squares method and time series interpolation.

4. The method according to claim 1, characterized in that: The step 3 comprises: Step 31, respectively obtaining the monitoring data of different data structures in different databases through a data access component; Step 32, uniformly convert the data format through format conversion, mapping conversion and / or conversion conversion methods to obtain the target data in the preset format, wherein format conversion is used for the situation where the data structure of the monitoring data and the data structure of the corresponding target data are different; mapping conversion is used for the situation where the fields of the data table of the monitoring data and the data table of the corresponding target data are different; conversion conversion is used for the situation where the monitoring data is data obtained by direct monitoring and needs to be converted by calculation.

5. The method according to claim 1, characterized in that: The Flink cluster consists of 4 machines, including 1 master node and 3 slave nodes, wherein the master node is the JobManager node in the Flink cluster, and the three slave nodes are the TaskManager nodes in the Flink cluster. Each node is connected through Gigabit Ethernet, and the operation mode between the nodes is Yarn mode.

6. The method according to claim 1, characterized in that The resource performance data includes CPU utilization and memory usage.

7. The method according to claim 6, characterized in that The calculation formula of the node load data is: L j =θ c L cj +θ m L mj Among them, L j is the load data of the jth node, θ c and θ m is the parameter weight coefficient, θ c +θ m =1,L cj is the CPU utilization, L mj is the memory usage, j is a positive integer from 1 to J, and J is the total number of nodes in the Flink cluster; The calculation formula for the average load of the Flink cluster is: Wherein, L is the average load.

8. The method according to claim 1, characterized in that: The step 6 comprises: Step 61: Collect a cluster load time series data set with a preset time interval and a length of N, and perform a second preprocessing on the cluster load time series data set; Step 62: Divide the preprocessed data set into a training set and a test set; Step 63: Use the deep learning framework TensorFlow to build an LSTM model, determine the parameters that need to be optimized, and determine the optimization range of each parameter, wherein the parameters that need to be optimized include the number of neurons, the learning rate, and the number of training iterations; Step 64: Use the training set to train the LSTM model; Step 65: Use the test set to evaluate the trained LSTM model; Step 66: Input the load data of each node into the trained LSTM model to obtain the load prediction result within a preset time.

9. The method according to claim 8, characterized in that The second preprocessing includes removing outliers and / or filling missing values.

10. The method according to claim 1, characterized in that In the particle swarm optimization algorithm, a linear decreasing inertia weight is used.

Images