Parallel Deep Forest Algorithm Based on Spark and NRSCA Strategies for Regional Landslide Hazard Assessment

Abstract

This paper proposes a parallel deep forest algorithm for regional landslide hazard assessment using Spark and NRSCA strategies. The algorithm comprises the following steps: S1: inputting a geographic base map of potential landslide areas into the parallel deep forest algorithm, filtering irrelevant features from the map to obtain a candidate feature set, filtering out redundant features from the candidate feature set to form a high-quality feature set; S2: sending the obtained high-quality feature set to a multi-granularity scanning stage for processing; S3: parallelizing the training of each layer of the cascade forest model using Spark; and S4: verifying the effectiveness of the parallel deep forest algorithm. The algorithm proposed in this paper significantly improves both operational efficiency and model accuracy. Furthermore, the knowledge mined by this method can provide significant benefits in the financial and medical fields.

Description

Technical Field

[0001] The present invention relates to the field of big data mining, and in particular to a Spark and NRSCA strategy parallel deep forest algorithm in a big data environment. Background Art

[0002] Landslides are a common sudden geological disaster in mountainous areas. They refer to the phenomenon that slopes become unstable due to external factors (such as rainfall, earthquakes, and human engineering activities), causing the rock and soil to slide down along certain weak surfaces (zones). They often bury villages, destroy roads, damage farmland, and block rivers. In some areas, they even induce secondary disasters such as mudslides in high mountain canyons. For a long time, they have threatened the safety of life and property of the general public, restricted regional economic development, and brought difficulties to land space planning and control. Regional landslide hazard assessment is to evaluate the possibility of a landslide geological disaster of a specific scale occurring in a certain area within a certain time period. It can provide local governments with corresponding prevention and control measures and provide a basic basis for geological disaster prevention and control, urban planning, and land use.

[0003] In today's big data era, many scholars have begun to mine big data on historical landslide data and related inducing factors, and conduct research on regional landslide hazard assessment with the help of deep integration of artificial intelligence and geographic information technology (GIS). In 2014, Christos Polykretis et al. combined neural networks with traditional data statistics to conduct a landslide hazard assessment on the Peloponnese Peninsula in Greece using elevation, slope, aspect, surface cover, lithology, and distance from water systems, roads, and faults as statistical indicators; in 2015, Ahmed Mohamed Youssef et al. used the random forest method and integrated the study area's elevation distribution, slope aspect, slope, cultivated area distribution, terrain profile index, lithology, distance from water systems, and rainfall to conduct a landslide hazard assessment and map compilation for Asir Province, Saudi Arabia; in 2017, Kaixiang Zhang et al. used the random forest decision tree method to integrate geographic information technology to conduct a landslide hazard assessment on the central area of ​​the Three Gorges in China, combining indicators such as landform, geology, hydrology, surface cover, and environmental inducing factors; in 2020, Deliang Sun et al. used the random forest method to combine 22 indicators including geomorphology, geological conditions, environmental factors and human engineering activities to evaluate the landslide hazard in Fengjie County and Wushan County in the Three Gorges region of China.

[0004] In summary, although machine learning, especially random forest fusion geographic information technology, has achieved fruitful results in the field of regional landslide hazard assessment, there are still some shortcomings: (1) In the geographic information software ArcGis, a geographic base map with a resolution of 25m*25m-1000m*1000m (depending on the data source and the evaluation accuracy requirements) is usually selected for the study area to extract the geological and geomorphological parameters and characteristics of all slope units one by one to assist experts in landslide hazard assessment. If a resolution of about 25m*25m is used, although the accuracy is sufficient to reflect the characteristics of a single slope unit, the data extraction process will generate tens of millions or even more raster data, and the computing performance of a single processor system cannot meet the required processing speed. If the extraction is performed based on a resolution of 1000m*1000m, it is easy to cause the coverage area to be too large and include multiple slopes, which is obviously not in line with the actual situation of a single slope itself. (2) Machine learning requires that the model accumulate knowledge in the process of continuous training and learning to improve effective knowledge identification. In addition, it must have the ability to generalize knowledge and reason. Traditional machine learning methods cannot retain the memory of the previous task while completing the current training task, and it is difficult to form the reuse and generalization of training knowledge. Therefore, how to design a method that can not only improve the prediction accuracy of large-scale landslide hazard assessment and make its processing speed meet practical application, but also enable the learning process to have knowledge reuse and generalization capabilities is of great significance.

[0005] Deep Forest is a forest-based ensemble learning method with a low number of hyperparameters and excellent representation learning capabilities. It is widely used in various fields, including image recognition, fault diagnosis, and indicator prediction. However, with the advent of the big data era, the amount of data and the number of features in various fields are growing exponentially. This increase in data volume and features significantly increases the training cost of Deep Forest, leading to a decline in training effectiveness. Therefore, a Deep Forest algorithm suitable for processing big data is needed to meet the needs of large-scale landslide hazard assessment. Summary of the Invention

[0006] The present invention aims to at least solve the technical problems existing in the prior art. To achieve the above-mentioned purpose of the present invention, the present invention provides a parallel deep forest algorithm for regional landslide hazard assessment using Spark and NRSCA strategies, comprising the following steps:

[0007] S1, input the geographic base map of potential landslide areas into the parallel deep forest algorithm, and filter out irrelevant features from it to obtain a candidate feature set, filter out redundant features from the candidate feature set, and form a high-quality feature set;

[0008] S2, sends the obtained high-quality feature set to the multi-granularity scanning stage for processing;

[0009] S3, combined with Spark, parallelizes each layer of the cascade forest model to obtain the landslide results.

[0010] In a preferred embodiment of the present invention, S1 specifically comprises the following steps:

[0011] S11, first calculate the Fisher Score (FS f ) and SU δ (f,C); then according to SU δ (f,C) and FS f Calculate the feature segmentation parameter FSP of each feature in the original feature set and sort them in descending order according to the size of FSP; finally, divide the original feature set into two parts: candidate feature set and irrelevant feature set according to the FSP value from high to low. The calculation formula of the feature segmentation parameter FSP is as follows;

[0012] FSP=FS f *SU δ (f,C) (1)

[0013] FSP represents the feature segmentation parameter;

[0014] FS f Indicates the value of Fisher Score of feature f;

[0015] SU δ (f, C) represents the correlation between feature f and label set C;

[0016] * indicates multiplication;

[0017] in,

[0018]

[0019] FS f Indicates the value of Fisher Score of feature f;

[0020] k represents the number of samples;

[0021] n c Indicates the number of samples of category c;

[0022] μ fc represents the mean of feature f in category c;

[0023] μ f represents the mean of feature f;

[0024] Represents the square of the variance of feature f in category c;

[0025]

[0026] SU δ(f, C) represents the correlation between feature f and label set C;

[0027] NH δ (f; C) represents the neighborhood mutual information between the current feature f and the label set C;

[0028] NH δ (f) represents the neighborhood information entropy of the current feature f;

[0029] NH δ (C) represents the neighborhood information entropy of the current label set C;

[0030] S12 proposes to filter the candidate feature set through the redundancy coefficient RF, further screen out a large number of redundant features, and obtain a high-quality feature set. The calculation formula of the redundancy coefficient RF is as follows:

[0031]

[0032] f i Indicates the first characteristic of the hypothesis;

[0033] f j Indicates the second characteristic of the hypothesis;

[0034] RF i,j represents f i With f j The redundancy coefficient of

[0035] SU δ (f i ,f j ) represents the measurement of the current feature f i and the second feature f j relevance;

[0036] NH δ (f; C) represents the neighborhood mutual information between the current feature f and the label set C.

[0037] In a preferred embodiment of the present invention, S2 specifically comprises the following steps:

[0038] S21, random scan: In the feature preprocessing stage, a feature set S is obtained, whose size is x and the number of categories is y. H features are randomly extracted from S to form a feature segment. This extraction is repeated b times to generate a total of b feature segments. Each feature segment is imported into the random forest for training to obtain b y-dimensional class vectors.

[0039] S22, isometric extraction: Divide the feature set S into b feature segments, each of size Import each feature fragment into the completely random forest for training, and also obtain b y-dimensional class vectors;

[0040] S23, initialize the class vector: repeat the above two stages z times, concatenate the class vectors obtained after z times of training, and finally obtain the original class vector E of 2zby dimension.

[0041] In a preferred embodiment of the present invention, S3 specifically comprises the following steps:

[0042] S31, Parallel Construction of Cascade Forests: Using the Spark framework to build parallel deep forests, we proposed a feature filtering mechanism for FFM-II. This mechanism removes non-critical features based on the accuracy of cascade forest training and balances the dimensions of the enhanced and original class vectors.

[0043] S32, Load Balancing: A TSM-SCA task scheduling mechanism was designed, and the SCA algorithm was used to calculate the optimal task allocation plan. The Spark cluster then allocated tasks according to the plan.

[0044] In a preferred embodiment of the present invention, S31 specifically includes the following steps:

[0045] S311, filter features; first calculate the importance index G(f) of each feature according to the training accuracy of the current layer cascade forest, and sort the features in the original class vector from low to high according to the value of G(f); then propose an adaptive scale coefficient Q DF , calculate the adaptive scale factor Q according to the number of features of the current original class vector DF ;Finally, according to Q DF The sorted original class vector E is divided into non-key class vectors and valid class vectors, and the non-key class vectors are placed into the non-key class vector set R * , where the formula for the importance index G(f) is as follows:

[0046]

[0047] G(f) represents the importance index of feature f of the current level cascade forest;

[0048] μ i represents the decision tree weight;

[0049] ε j represents the weight of the j-th sub-forest;

[0050] s represents the number of sub-forests contained in each cascade forest;

[0051] r represents the number of decision trees contained in the sub-forest;

[0052] G i (f) represents the importance index of feature f of the i-th decision tree in the j-th subforest;

[0053]

[0054] μ i represents the decision tree weight;

[0055] P i represents the accuracy of the i-th decision tree in the j-subforest;

[0056] r represents the number of decision trees contained in the sub-forest;

[0057]

[0058] ε j represents the sub-forest weight;

[0059] Q j represents the accuracy of the j-th sub-forest;

[0060] s represents the number of sub-forests contained in each cascade forest;

[0061] Among them, the adaptive proportional coefficient Q DF The calculation formula is as follows:

[0062]

[0063] Q DF Represents the adaptive scale coefficient;

[0064] λ represents the number of layers in the cascade forest;

[0065] N λ represents the number of features of the original vector of the λth layer cascade forest;

[0066] S312, merging class vectors: First, concatenate the enhanced class vectors obtained from all previous layers of cascade forest training to obtain a total enhanced class vector D; then concatenate the total enhanced class vector D with the original class vector E' of the next layer of cascade forest; finally, delete the non-critical class vector set R from the concatenated class vectors * The features contained in are used to obtain the class vector ED that is input to the next layer of cascade forest.

[0067] In a preferred embodiment of the present invention, S32 specifically includes the following steps:

[0068] S321, initialize SCA population: initialize the population size to k, X m =(x m1 ,x m2 ,.....,x mN ) represents the mth (m=1,2,.....,k) individual, and the search space dimension is set to N according to the number of cluster nodes;

[0069] S322, update the position of the individual: propose a position update function W(x) based on the inertia factor ω to replace formula (10), so as to improve the global search ability of the individual and accelerate the convergence of the algorithm; the formula included is as follows:

[0070]

[0071] W() represents the position update function;

[0072] represents the updated position of individual h on dimension l;

[0073] ω(t) represents the inertia factor;

[0074] represents the original position of individual h on dimension l;

[0075] r1 represents the updated amplitude control parameter;

[0076] r2 represents a random number that obeys a uniform distribution, r2∈[0,2π];

[0077] r3 represents a random number that obeys uniform distribution, r3∈[-2,2];

[0078] r4 represents a random number that obeys uniform distribution, r4∈[0,1];

[0079] p gl Indicates the l-dimensional position of the current global optimal solution;

[0080] The calculation formula of the inertia factor ω(t) is as follows:

[0081]

[0082] ω min Indicates the inertia factor at the end of the iteration, that is, the minimum value;

[0083] ω max represents the initial inertia factor, i.e. the maximum value;

[0084] t represents the current iteration number;

[0085] T represents the maximum number of iterations;

[0086] Represents the ratio of t to T

[0087] S323, update the amplitude control parameter r1: In formula (11), r1 is linearly decreasing, and the decreasing speed is the same in the early and late stages, resulting in insufficient global search in the early stage and the inability of the algorithm to converge quickly in the late stage. In order to obtain better stability and higher optimization ability, a curve adaptive amplitude adjustment strategy based on the tangent function is proposed for the control parameter r1 to ensure the quality of the solution. The formula is as follows:

[0088]

[0089] r1(t) represents the amplitude control parameter;

[0090] a represents a constant;

[0091] t represents the current iteration number;

[0092] T represents the maximum number of iterations;

[0093] represents the ratio of t to T;

[0094] S324, determine whether the current optimal individual meets the solution conditions: if the conditions are met or the set number of iterations reaches the upper limit, the current optimal individual P is output. g , does not satisfy the repetition S322-S323;

[0095] S325, task allocation: The cluster allocates tasks according to the obtained task scheduling plan P g =(p g1 ,p g2 ,....,p gN ) to distribute tasks and achieve load balancing of the cluster.

[0096] In summary, due to the adoption of the above technical solutions, the present invention can address the problems of the parallel deep forest algorithm in the big data environment, such as many redundant data features, low feature utilization at both ends of multi-granularity scanning, and slow convergence speed. The present invention proposes an improved parallel deep forest algorithm based on neighborhood rough sets - PDF-SNRSCA. The main work of the algorithm is as follows: (1) In the feature selection stage, a feature selection strategy based on neighborhood rough sets and Fisher Score (FS-NRS) is proposed. By measuring the correlation and redundancy of features, features with large information content are screened out to solve the problem of excessive redundant and irrelevant features. (2) In the multi-granularity scanning stage, a scanning strategy for random selection and equidistant extraction (S-RSEE) is proposed to replace the original multi-granularity scanning strategy to solve the problem of low feature utilization at both ends of multi-granularity scanning. (3) During the model parallel training phase, a feature filtering mechanism based on importance index (FFM-II) was proposed to balance the dimensions of the enhanced class vector and the original class vector, solving the problem of slow model convergence. At the same time, a task scheduling mechanism based on SCA (TSM-SCA) was designed to redistribute tasks, ensure cluster load balancing, and improve the efficiency of cascade forest parallelization. The algorithm proposed in this paper has significantly improved both operational efficiency and model accuracy. In addition, the knowledge mined by this method can provide great help in the financial and medical fields.

[0097] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0098] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0099] Figure 1 It is a schematic diagram showing the experimental results of the present invention.

[0100] Figure 2 It is a schematic diagram showing the accuracy F-measure of the present invention. DETAILED DESCRIPTION

[0101] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0102] 1. Parallel feature extraction

[0103] Currently, in the parallel deep forest algorithm under the big data environment, there is a problem of too many redundant features in the model training process. To solve this problem, a feature selection strategy called FS-NRS is proposed. This strategy mainly consists of two steps: (1) Filtering irrelevant features: The feature segmentation parameter FSP is proposed to segment the original feature set, filter out a large number of irrelevant features, and obtain a candidate feature set. (2) Filtering redundant features: The redundant coefficient RF is proposed to filter the candidate feature set, filter out a large number of redundant features, and obtain a high-quality feature set.

[0104] 1.1 Screening out irrelevant features

[0105] In order to accurately filter out a large number of irrelevant features in the original feature set, the feature segmentation parameter FSP is proposed to divide the original feature set into irrelevant feature set and candidate feature set. The division process is as follows: First, calculate the Fisher Score (FS f ) and SU δ (f,C); then according to SU δ (f,C) and FS f Calculate the feature segmentation parameter FSP of each feature in the original feature set and sort them in descending order according to the size of FSP; finally, divide the original feature set into two parts: candidate feature set and irrelevant feature set according to the FSP value from high to low.

[0106] Theorem 1 (Feature Segmentation Parameter FSP): Given a sample set with k categories, the feature segmentation parameter FSP of feature f is:

[0107] FSP=FS f *SU δ (f,C) (1)

[0108] FSP represents the feature segmentation parameter;

[0109] FS f Indicates the value of Fisher Score of feature f;

[0110] SU δ (f, C) represents the correlation between feature f and label set C;

[0111] * indicates multiplication;

[0112] in,

[0113]

[0114] FS f Indicates the value of Fisher Score of feature f;

[0115] k represents the number of samples;

[0116] n c Indicates the number of samples of category c;

[0117] μ fc represents the mean of feature f in category c;

[0118] μ f represents the mean of feature f;

[0119] Represents the square of the variance of feature f in category c;

[0120]

[0121] SU δ (f, C) represents the correlation between feature f and label set C;

[0122] NH δ (f; C) represents the neighborhood mutual information between the current feature f and the label set C;

[0123] NH δ (f) represents the neighborhood information entropy of the current feature f;

[0124] NH δ (C) represents the neighborhood information entropy of the current label set C.

[0125] Proof: According to the definition of neighborhood symmetric uncertainty, SU δ (f, C) can measure the correlation between the current feature f and the label set C. When SU δ When (f, C) is larger, it indicates that the correlation between the current feature f and the label set C is greater; according to the definition of FisherScore, FS f It can measure the importance of the current feature f, FS f The larger the value, the more important the current feature f is. FSP is set as the product of neighborhood symmetric uncertainty and Fisher Score, taking into account both feature correlation and feature importance. When the selected feature has a high correlation with the label and the feature importance is higher, the larger the FSP value is, and the more important features are selected. Therefore, FSP can be used as feature segmentation.

[0126] 1.2 Filtering redundant features

[0127] After the initial filtering of irrelevant features, there are still a lot of redundant features in the candidate feature set. Therefore, the redundancy coefficient RF is proposed to filter the candidate feature set, further screen out a large number of redundant features, and obtain a high-quality feature set. The process is as follows: First, calculate the neighborhood mutual information NH between the current feature f and the label set C δ (f i ; C) Neighborhood uncertainty SU between features δ (f i ,f j );Then use NH δ (f i ; C) and SU δ (f i ,f j ) Calculate the redundancy coefficient RF between features, and further eliminate redundant features based on the comparison between the redundancy coefficient and the threshold to obtain a high-quality feature set.

[0128] Theorem 2 (Redundancy coefficient RF): It is known that there is a feature f i With f j , then the redundancy coefficient RF of the two features i,j for:

[0129]

[0130] f i Indicates the first characteristic of the hypothesis;

[0131] f j Indicates the second characteristic of the hypothesis;

[0132] RF i,j represents f i With f j The redundancy coefficient of

[0133] SU δ ( fi,fj ) represents the measurement of the current feature f i and the second feature f j relevance;

[0134] NH δ (f; C) represents the neighborhood mutual information between the current feature f and the label set C.

[0135] Proof: SU δ (f i ,f j ) is the neighborhood symmetric uncertainty between features. According to the neighborhood symmetric uncertainty, SU δ (fi ,f j ) can measure the degree of correlation between features, SU δ (f i ,f j ) is larger, indicating that the correlation between features is greater and the redundancy of information between features is greater; δ (f i ; C) is the neighborhood mutual information between features and labels. According to the neighborhood mutual information, NH δ (f i ; C) can measure the correlation between features and labels, NH δ (f i ; C) is larger, indicating that the amount of relevant information between the feature and the label is greater; when SU δ (f i ,f j ) is larger, NH δ (f i ; C) The smaller the value, the smaller the characteristic f i ,f j The greater the amount of overlapping information between features, the greater the i The smaller the correlation with label C, the more likely SU δ (f i ,f j ) / NH δ (f i ; C) The larger the ratio, the greater the redundancy between features. Therefore, the redundancy coefficient RF can measure the amount of redundant information of the features. The proof is complete.

[0136] 2. Multi-granularity scanning

[0137] The high-quality feature set obtained in the feature preprocessing stage is sent to the multi-granularity scanning stage for processing. However, traditional multi-granularity scanning has the problem of low feature utilization at both ends. To address this, the S-RSEE feature scanning strategy is proposed. The specific process of this strategy is as follows:

[0138] (1) Random scanning. In the feature preprocessing stage, a feature set S is obtained, whose size is x and the number of categories is y. H features are randomly extracted from S to form a feature segment. This extraction is repeated b times, generating a total of b feature segments. Each feature segment is introduced into the random forest training to obtain b y-dimensional class vectors.

[0139] (2) Equidistant extraction. Divide the feature set S into b feature segments, each of size Import each feature fragment into the completely random forest for training, and also obtain b y-dimensional class vectors.

[0140] (3) Initialize the class vector. Repeat the above two stages z times, concatenate the class vectors obtained after z training times, and finally obtain the original class vector E of 2zby dimension.

[0141] 3. Cascade Forest Parallel Training

[0142] To address the issues of slow model convergence and low efficiency of parallel training of cascade forests during parallel training, this paper uses Spark to parallelize the training of each layer of the cascade forest model. The process is divided into two stages: (1) Parallel construction of cascade forests: Using the Spark framework to build a parallel deep forest, a feature screening mechanism for FFM-II is proposed. Non-critical features are eliminated based on the accuracy obtained from cascade forest training, and the dimensions of the enhanced class vector and the original class vector are balanced. (2) Load balancing: A task scheduling mechanism called TSM-SCA is designed, and the SCA algorithm is used to calculate the optimal task allocation plan. The Spark cluster then allocates tasks according to the plan.

[0143] 3.1 Parallel Construction of Cascade Forests

[0144] When using Spark to build a parallel cascade forest, the dimension of the enhanced class vector obtained by cascade forest training is much lower than that of the original class vector, resulting in slow model convergence. Therefore, we proposed the FFM-II feature screening mechanism. This mechanism calculates the feature importance index based on the model's training accuracy, thereby eliminating some non-critical features and balancing the dimension of the enhanced class vector and the original class vector, thereby accelerating model convergence. The strategy process is as follows:

[0145] (1) Filtering features. First, calculate the importance index G(f) of each feature based on the training accuracy of the current layer cascade forest, and sort the features in the original class vector from low to high according to the value of G(f); then propose an adaptive scaling coefficient Q DF , calculate the adaptive scale factor Q according to the number of features of the current original class vector DF ;Finally, according to Q DF The sorted original class vector E is divided into non-key class vectors and valid class vectors, and the non-key class vectors are placed into the non-key class vector set R * .

[0146] (2) Merge class vectors. First, concatenate the enhanced class vectors obtained from all previous layers of cascade forest training to obtain the total enhanced class vector D; then concatenate the total enhanced class vector D with the original class vector E' of the next layer of cascade forest; finally, delete the non-critical class vector set R from the concatenated class vector. * The features contained in are used to obtain the class vector ED that is input to the next layer of cascade forest.

[0147] Theorem 3 (Importance Index G(f)) Assume that in the current level forest, the decision tree weight is μi , the sub-forest weight is ε j , the importance of feature f in the i-th decision tree in the j-th subforest is G i (f), then in the current level cascade forest, the importance index of feature f is:

[0148]

[0149] G(f) represents the importance index of feature f of the current level cascade forest;

[0150] μ i represents the decision tree weight;

[0151] ε j represents the weight of the j-th sub-forest;

[0152] s represents the number of sub-forests contained in each cascade forest;

[0153] r represents the number of decision trees contained in the sub-forest;

[0154] G i (f) represents the importance index of feature f of the i-th decision tree in the j-th subforest;

[0155]

[0156] μ i represents the decision tree weight;

[0157] P i represents the accuracy of the i-th decision tree in the j-subforest;

[0158] r represents the number of decision trees contained in the sub-forest;

[0159]

[0160] ε j represents the sub-forest weight;

[0161] Q j represents the accuracy of the j-th sub-forest;

[0162] s represents the number of sub-forests contained in each layer of the cascade forest.

[0163] Among them, P i is the accuracy of the i-th decision tree in the j-th sub-forest, Q j is the accuracy of the j-th sub-forest.

[0164] Proof: Assume that each cascade forest contains s sub-forests, and each forest contains r decision trees. The accuracy of the i-th decision tree in the j-th sub-forest is P i, the accuracy of the j-th sub-forest is Q j , the accuracy can measure the classification ability of decision trees and sub-forests. i and The ratio of represents the normalized weight of the prediction accuracy of a single decision tree and the total accuracy of the sub-forest. The larger the value, the stronger the classification ability of the decision tree and the more important it is in the sub-forest. Similarly, Q j and The ratio reflects the Q j The total accuracy of the cascade forest at this level The weight of G is larger, and the larger its value is, the stronger the classification ability of the sub-forest is, and the more important it is in the cascade forest at this level. i (f) represents the importance of feature f in the i-th decision tree, G i The value of (f) can only measure the importance of the feature in the current decision tree. If you want to measure the importance of feature f in the current cascade forest, you need to calculate the importance of the feature in all decision trees in the current layer. Therefore, in the current cascade forest, the importance index of feature f is:

[0165]

[0166] G(f) represents the importance index of feature f of the current level cascade forest;

[0167] s represents the number of sub-forests contained in each cascade forest;

[0168] r represents the number of decision trees contained in the sub-forest;

[0169] ε j represents the sub-forest weight;

[0170] μ i represents the decision tree weight;

[0171] G i (f) represents the importance index of feature f of the i-th decision tree in the j-th sub-forest.

[0172] The proof is complete.

[0173] Theorem 4 (Adaptive proportional coefficient Q DF ) Assume that in the λth layer cascade forest, the number of features of the original class vector is N λ , then the adaptive scaling coefficient Q of the λth layer DF for:

[0174]

[0175] Q DF Represents the adaptive scale coefficient;

[0176] λ represents the number of layers in the cascade forest;

[0177] N λ Represents the number of features of the original vector of the λth layer cascade forest.

[0178] Proof: When λ>1, the number of features of the original vector of the λth layer cascade forest is N λ ,make at this time but When the number of features N in the λ+1th layer λ When unchanged, let at this time When the number of features in the λ+1th layer is N λ+1 At this time Since the function f(x)=1 / x takes the value (0,1) in [1,+∞), then make This will cause the proportional coefficient to change dynamically with the number of layers and features, and gradually decrease as the number of layers increases.

[0179] In summary, Q DF It is a coefficient that can dynamically adjust the division ratio according to the number of layers and features of the cascade forest, and the more layers of the cascade forest, the smaller the division ratio.

[0180] 3.2 Load Balancing

[0181] To address the problem of uneven cluster load caused by differences in the computing power of Spark nodes, which leads to low efficiency in cascade forest parallelization, we designed the TSM-SCA task scheduling mechanism. This mechanism uses the SCA algorithm to find the optimal scheduling solution and redistribute tasks to achieve a load-balanced state among nodes, thereby improving parallelization efficiency. The specific steps are as follows:

[0182] (1) Initialize the SCA population. The initial population size is k, X m =(x m1 ,x m2 ,.....,x mN ) represents the mth (m=1,2,.....,k) individual, and the search space dimension is set to N according to the number of cluster nodes.

[0183] (2) Update the position of the individual. A position update function W(x) based on the inertia factor ω is proposed to replace Equation (10) to improve the global search ability of the individual and accelerate the convergence of the algorithm.

[0184] (3) Update the amplitude control parameter r1. In formula (11), r1 decreases linearly, and the decrease rate is the same in the early and late stages. This leads to insufficient global search in the early stage and the inability of the algorithm to converge quickly in the later stage. To achieve better stability and higher optimization capability, a curve-adaptive amplitude adjustment strategy based on the tangent function is proposed for the control parameter r1 to ensure the quality of the solution.

[0185] (4) Determine whether the current optimal individual meets the solution conditions. If the conditions are met or the set number of iterations reaches the upper limit, the current optimal individual P is output. g , does not satisfy the repetition (2)-(3).

[0186] (5) Task allocation. The cluster allocates tasks according to the obtained task scheduling scheme P. g =(p g1 ,p g2 ,....,p gN )(g1, g2, ..., gN represent the positions of the optimal individuals in each dimension) to perform task allocation and achieve load balancing of the cluster.

[0187] Theorem 5 (position update function W()): Assume is the original position of individual h in dimension l, is the updated position of individual h in dimension l, p gl is the l-dimensional position of the current global optimal solution, ω(t) is the inertia factor, and the position update function is (there are two position update functions, and the size of r4 is needed to determine whether to use the upper update or the lower update):

[0188]

[0189] W() represents the position update function;

[0190] represents the updated position of individual h on dimension l;

[0191] ω(t) represents the inertia factor;

[0192] represents the original position of individual h on dimension l;

[0193] r1 represents the updated amplitude control parameter;

[0194] r2 represents a random number that obeys a uniform distribution, r2∈[0,2π];

[0195] r3 represents a random number that obeys uniform distribution, r3∈[-2,2];

[0196] r4 represents a random number that obeys uniform distribution, r4∈[0,1];

[0197] p gl Indicates the l-dimensional position of the current global optimal solution;

[0198]

[0199] ω(t) represents the inertia factor;

[0200] ω min Indicates the inertia factor at the end of the iteration, that is, the minimum value;

[0201] ω max represents the initial inertia factor, i.e. the maximum value;

[0202] t represents the current iteration number;

[0203] T represents the maximum number of iterations;

[0204] It represents the ratio of t to T.

[0205] Where T is the maximum number of iterations, ω max is the initial inertia factor, i.e. the maximum value; ω min is the inertia factor at the end of the iteration, that is, the minimum value.

[0206] Proof: According to the iteration theorem of inertia factor, in the early stage of iteration, a larger inertia factor can improve the global search ability, and in the late stage of iteration, a smaller inertia factor can enhance the local development ability and accelerate the convergence of the algorithm. The ratio of the current number of iterations t to the maximum number of iterations T Indicates the time period of the current iteration, the ratio of t to T is between [0,1], because the function exist The interval decreases slowly at first, then decreases rapidly, and when When ω→ω max , When ω→ω min , so the inertia factor ω takes a larger value in the early stage and a smaller value in the later stage. ω(t) satisfies the iteration theorem of the inertia factor. Therefore, using W(x) can improve the global search capability and accelerate the convergence of the algorithm. The proof is complete.

[0207] Theorem 6 (r1 adaptive amplitude adjustment strategy): Assuming T is the maximum number of iterations and a is a constant, then at t iterations, the control parameter r1 is:

[0208]

[0209] r1(t) represents the amplitude control parameter;

[0210] a represents a constant;

[0211] t represents the current iteration number;

[0212] T represents the maximum number of iterations;

[0213] It represents the ratio of t to T.

[0214] Proof: The ratio t / T of the current iteration number t and the maximum iteration number T represents the time period of the current iteration. The ratio of t to T is between [0,1], because the function exist The interval decreases slowly at first and then rapidly, so the deceleration of r1∈[0,a] starts slowly and then accelerates. The slow decrease of r1 in the early stages ensures a greater number of iterations than the original SCA algorithm, which significantly increases global search power and helps search for the optimal solution within a larger space. Later, the accelerated decrease of r1 accelerates algorithm convergence, improving both the accuracy and speed of SCA's optimization. Therefore, r1(t) can enhance global search power, accelerate convergence, and improve solution accuracy. Complete proof.

[0215] 4. Verification of the effectiveness of the PDF-SNRSCA algorithm

[0216] In order to verify the effectiveness and feasibility of the PDF-SNRSCA algorithm, the Yan'an City and Yan'an Ansai District were rasterized to obtain 60,185,000 and 5,600,000 grid cells, respectively. Each grid cell has 7 influencing factors (dimensions). They are called the Yan'an City and Yan'an Ansai District datasets, respectively. The PDF-SNRSCA algorithm was applied to the above two datasets to verify the performance of the algorithm.

[0217] 4.1 Parallel Analysis of PDF-SNRSCA Method

[0218] In order to verify the feasibility of the PDF-SNRSCA algorithm in a big data environment, the acceleration ratio is used as an evaluation indicator to conduct experiments on two data sets: Yan'an City and Ansai District of Yan'an City. In order to further ensure the accuracy of the experiment, the acceleration ratio is calculated based on the average result after running 10 times. The experimental results are as follows: Figure 1 shown.

[0219] Depend on Figure 1As can be seen, the algorithm's speedup ratios on both the Yan'an City and Ansai District datasets gradually increase with the number of nodes. The speedup ratio increases slightly when the algorithm is running on two nodes. This is because the large amount of data allocated to each node does not significantly improve parallel performance. However, the speedup ratio increases linearly with the number of nodes. When the number of nodes increases to 6, the algorithm achieves a speedup ratio of 4.56 when processing the Yan'an City dataset, a 3.56 improvement over a single node. When processing the Ansai District dataset, the speedup reaches 4.12, a 3.12 improvement over a single node. The main reasons for this result are: (1) When the number of nodes is small, the overall running time of the algorithm is mainly determined by the communication time between nodes. The limited computing time reduced by parallel computing cannot play a dominant role, resulting in a relatively small acceleration of the algorithm at this time; (2) When the number of nodes is large, the PDF-SNRSCA algorithm uses the FS-NRS strategy to filter out a large number of redundant and irrelevant features in the original data set, reducing the calculation of redundant and irrelevant features, thereby improving the algorithm's acceleration ratio; (3) In the model parallel training stage, the PDF-SNRSCA algorithm uses the FFM-II mechanism to balance the dimensions of the enhanced vector and the original vector, accelerating the model convergence speed. At the same time, it uses the TSM-SCA mechanism to find the optimal scheduling scheme and achieve load balancing, thereby improving the parallel computing efficiency of each node. Therefore, the performance improvement of the PDF-SNRSCA algorithm on the parallel computing platform will be more significant as the number of nodes increases, which also shows that the PDF-SNRSCA algorithm has good feasibility in the big data environment.

[0220] 4.2 Classification Accuracy Analysis of PDF-SNRSCA Method

[0221] In order to analyze the classification accuracy of the PDF-SNRSCA algorithm, experiments were conducted on two data sets, Yan'an City and Ansai District of Yan'an City. The results of the algorithm's accuracy F-measure are as follows: Figure 2 shown.

[0222] from Figure 2It can be seen that the classification accuracy of the PDF-SNRSCA algorithm on both datasets reached over 85%. On the Yan'an City dataset, the algorithm's F-measure value reached 88.5%; on the Yan'an City Ansai District dataset, the algorithm's F-measure value reached 87.2%. The main reasons for this result are: (1) The PDF-SNRSCA algorithm designed the FS-NRS strategy, which filtered the features by measuring the relevance and redundancy of the features, thereby improving the classification ability of the model; (2) The PDF-SNRSCA algorithm designed the S-RSEE strategy, which made the features selected with the same probability, improving the utilization rate of the features at both ends and thus improving the accuracy of the algorithm; (3) The PDF-SNRSCA algorithm used the FFM-II strategy to filter out non-critical features, which also helped to improve the accuracy of the algorithm. Therefore, from the above comparison results, it can be seen that the PDF-SNRSCA algorithm has better classification accuracy on the two datasets.

[0223] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A parallel deep forest method based on Spark and NRSCA strategies for regional landslide hazard assessment, characterized by: The following steps are involved: S1: Input the geographic base map of potential landslide areas into the improved parallel deep forest algorithm, filter out irrelevant features, obtain a candidate feature set, filter out redundant features from the candidate feature set, and form a high-quality feature set. S1 specifically includes the following steps: S11, first calculate the Fisher Score of each feature ( )and ; then according to and Calculate the feature segmentation parameters for each feature in the original feature set , and according to Sort by size in descending order; finally, Value, from high to low, divides the original feature set into two parts: candidate feature set and irrelevant feature set, among which the feature segmentation parameter The calculation formula is as follows; (1) Represents feature segmentation parameters; Representation characteristics The value of Fisher Score; Represents measurement characteristics and tag collection relevance; S12, propose and pass the redundancy coefficient Filter the candidate feature set to screen out a large number of redundant features and obtain a high-quality feature set; S2, sends the acquired high-quality feature set to the multi-granularity scanning stage for processing; S2 specifically includes the following steps: S21, random scanning: feature preprocessing stage to obtain feature set , whose size is , the number of categories is ,from Randomly select Features form a feature fragment, and repeated extraction times, a total of feature fragments, each feature fragment is imported into the random forest for training, and the indivual -dimensional class vector; S22, isometric extraction: feature set Divide into Characteristic fragments, each of size , import each feature fragment into the completely random forest for training, and we also get indivual -dimensional class vector; S23, initialize class vector: repeat the above two stages Second, The class vectors obtained after training are concatenated and finally obtained The original class vector of dimension ; S3, combined with Spark, parallelizes each layer of the cascade forest model to obtain landslide results; S3 specifically includes the following steps: S31, Parallel Construction of Cascade Forests: Use the Spark framework to build a parallel deep forest, combined with the feature filtering mechanism of FFM-II, remove non-critical features based on the accuracy obtained from cascade forest training, and balance the dimensions of the enhanced class vector and the original class vector; S32, load balancing: Through the TSM-SCA task scheduling mechanism, the SCA algorithm is used to calculate the optimal task allocation plan, and then the Spark cluster allocates tasks according to the plan.

2. The parallel deep forest method for regional landslide hazard assessment based on Spark and NRSCA strategies according to claim 1, characterized in that: in: (2) Representation characteristics The value of Fisher Score; Indicates the number of samples; Indicates the category The number of samples; Representation characteristics In category The mean of Representation characteristics The mean of Representation characteristics In category The square of the variance; (3) Represents measurement characteristics and tag collection relevance; Indicates the current feature and tag collection Neighborhood mutual information; Indicates the current feature Neighborhood information entropy; Represents the current tag set Neighborhood information entropy; Redundancy coefficient The calculation formula is as follows: (4) Indicates the first characteristic of the hypothesis; Indicates the second characteristic of the hypothesis; express and The redundancy coefficient of Indicates the measurement of current features and the second feature relevance; Indicates the current feature and tag collection Neighborhood mutual information.

3. The parallel deep forest method for regional landslide hazard assessment based on Spark and NRSCA strategies according to claim 1, characterized in that: S31 specifically includes the following steps: S311, filter features; first calculate the importance index of each feature based on the training accuracy of the current layer cascade forest , and according to The value of sort sorts the features in the original class vector from low to high; the adaptive scale coefficient is calculated according to the number of features of the current original class vector ; Finally, according to the original class vector whose values ​​will be sorted Divide into non-key class vectors and valid class vectors, and put the non-key class vectors into the non-key class vector set , where the importance index The formula is as follows: (5) Represents the characteristics of the current level cascade forest Importance index; represents the decision tree weight; Indicates the Weight of each subforest; Indicates the number of sub-forests contained in each layer of cascade forest; Indicates the number of decision trees contained in the sub-forest; Indicates the The first Characteristics of a decision tree Importance index; (6) represents the decision tree weight; express The first in the forest The accuracy of the decision tree; Indicates the number of decision trees contained in the sub-forest; (7) represents the sub-forest weight; Indicates the The accuracy of the sub-forest; Indicates the number of sub-forests contained in each layer of cascade forest; Among them, the adaptive scale coefficient The calculation formula is as follows: (9) Represents the adaptive scale coefficient; Indicates the number of layers in the cascade forest; Indicates the The number of features of the original vector of the layer cascade forest; S312, merge class vectors: First, concatenate the enhanced class vectors obtained from all previous layers of cascade forest training to obtain the total enhanced class vector ; Then the total enhanced class vector The original class vector of the next layer of cascade forest Perform splicing; finally, delete the non-key class vector set from the spliced ​​class vector The features contained in get the class vector input to the next layer of cascade forest .

4. The parallel deep forest method for regional landslide hazard assessment based on Spark and NRSCA strategies according to claim 1, characterized in that: S32 specifically includes the following steps: S321, initialize SCA population: initialize the population size to , Indicates the Individual, set the search space dimension according to the number of cluster nodes ; S322, Update individual position: Based on inertia factor Position update function To replace formula (10), in order to improve the individual's global search ability and accelerate the convergence of the algorithm; the formula included is as follows: (10) Represents the position update function; Represents an individual In dimension Update position on ; represents the inertia factor; Represents an individual In dimension The original position on Indicates updating of amplitude control parameters; represents a random number that follows a uniform distribution, ; represents a random number that follows a uniform distribution, ; represents a random number that follows a uniform distribution, ; Represents the current global optimal solution Dimensional position; Among them, the inertia factor The calculation formula is as follows: (11) Indicates the inertia factor at the end of the iteration, that is, the minimum value; represents the initial inertia factor, i.e. the maximum value; Indicates the current iteration number; Indicates the maximum number of iterations; express and The ratio of S323, update amplitude control parameters :In formula (11) It decreases linearly, and the decreasing speed is the same in the early and late stages, which leads to insufficient global search in the early stage and the inability of the algorithm to converge quickly in the later stage. In order to obtain better stability and higher optimization ability, the control parameters Formula (12) is used to ensure the quality of the solution; (12) represents the amplitude control parameter; represents a constant; Indicates the current iteration number; Indicates the maximum number of iterations; express and The ratio of S324, determine whether the current optimal individual meets the solution conditions: if the conditions are met or the set number of iterations reaches the upper limit, the current optimal individual is output , does not satisfy the repetition of S322 to S323; S325, task allocation: The cluster is based on the obtained task scheduling plan Perform task allocation to achieve cluster load balancing.

5. The parallel deep forest method for regional landslide hazard assessment based on Spark and NRSCA strategies according to claim 1, characterized in that: It also includes verification of the effectiveness of the improved parallel deep forest algorithm.

Images