A Parallel DCNN Classification Method for Landslide Hazard Assessment in Large Areas

Abstract

The present invention proposes a parallel DCNN classification method for large-area landslide hazard assessment, including the following steps: S1, grouping and merging the feature maps in the convolutional layer to achieve feature map compression, which solves the problem of excessive redundant feature maps during the training process. S2, during the parallel training of the model, the optimal approximate convolutional kernel is found within the given classification accuracy loss range to replace the original convolutional kernel, thereby reducing the invalid parameters in the convolutional kernel and overcoming the problem of low convolutional operation efficiency. S3, evenly distributing various intermediate data generated during the parallel training of the model effectively solves the problem of unbalanced node loads. The present invention can significantly improve the training efficiency of the DCNN model in a large data environment and at the same time exhibits good scalability on large datasets.

Description

Technical Field

[0001] The present invention relates to the technical fields of big data mining and GIS, and particularly relates to a parallel DCNN classification method for large-area landslide hazard assessment. Background Art

[0002] The frequent occurrence of landslides threatens life, property safety, and damages the natural environment and resources. The occurrence of a landslide refers to the phenomenon that a rock and soil mass is affected by external factors (such as rainfall, earthquake, human activities, etc.) and moves as a whole along a certain weak surface (zone). Especially when the rainy season comes, rainfall and human engineering activities induce landslides to occur simultaneously or successively at many points, and their harm is sufficient to seriously affect the local social and economic development. Regional landslide hazard assessment can reduce the losses caused by landslide disasters, and it is the core of landslide prevention and control work.

[0003] With the development of artificial intelligence and geographic information technology, many researchers have used these technologies to carry out large-area landslide prediction research. In the early stage, Nadim et al., Hong et al., and Cepeda et al. used the landslides in the World Landslide Database as sample data and respectively adopted the weighted linear combination method to conduct hazard assessments in regions such as the world and Indonesia; in 2013, Liu et al. input 9 landslide influencing factors such as slope, aspect, and curvature into a neural network algorithm to conduct landslide hazard assessment in the Chinese region; in 2014, Gunther et al. considered slope, terrain, geology, and land cover as landslide influencing factors and adopted the analytic hierarchy process to conduct hazard assessment in the European region; Kirschbaum and Stanley et al. respectively used different landslide influencing factors in 2016 and 2017 to predict the possibility of landslides occurring in Central America and the Caribbean region and depict their hazard assessment maps.

[0004] In summary, many achievements have been made in large-area landslide hazard assessment technology, but these methods still have some problems: in the ArcGis software, geological, geomorphic, and other data of slope units are extracted according to a resolution of 1000×1000, which obviously does not conform to the geological and geomorphic characteristics of slope units. Because in practical applications, a resolution of about 25×25 can well map the geological and geomorphic characteristics of a slope, and an ultra-large resolution of 1000×1000 will lead to the problem of low accuracy in landslide hazard assessment; if data is extracted with a resolution of about 25×25, tens of millions or even more raster cell data will be generated, and the computing performance of a single-processor system cannot meet the required processing speed. Therefore, it is of great significance to design a method that can not only improve the prediction accuracy of large-area landslide hazard assessment but also meet the processing speed of practical applications.

[0005] Deep Convolutional Neural Networks (DCNN) is an important classification algorithm in deep learning algorithms. It is widely used in small - area landslide hazard assessment, computer vision, image classification, speech recognition, semantic segmentation, natural language processing and other fields because of its good translational invariance, generalization ability and feature extraction ability. However, its computational complexity increases exponentially with the growth of data volume, and the training speed and various algorithm performances will also be severely reduced. Therefore, designing a DCNN method suitable for processing big data is necessary to meet the requirements of large - area landslide hazard assessment. Summary of the Invention

[0006] The present invention aims to at least solve the technical problems existing in the prior art, and particularly innovatively proposes a parallel DCNN classification method for large - area landslide hazard assessment.

[0007] To achieve the above object of the present invention, the present invention provides a parallel DCNN classification method for large - area landslide hazard assessment, including the following steps:

[0008] S0, obtain the landslide data to be processed;

[0009] S1, perform grouping and merging on the feature maps in the convolutional layer to achieve feature map compression, which solves the problem of too many redundant feature maps in the training process;

[0010] S2, in the model parallel training, find the optimal approximate convolution kernel to replace the original convolution kernel within a given classification accuracy loss range, so as to reduce the invalid parameters in the convolution kernel, and then overcome the problem of low convolution operation efficiency;

[0011] S3, evenly distribute various intermediate data generated during the model parallel training process, effectively solving the problem of unbalanced node load.

[0012] S4, obtain the landslide result according to the processing in steps S1 - S3.

[0013] Further, the S1 includes:

[0014] S1 - 1, feature map reconstruction: re - order the input feature map set G, and reconstruct the feature map into two - part feature map sets containing more effective information through singular value decomposition;

[0015] S1 - 2, redundant feature map filtering: filter out the redundant feature maps existing in the feature maps according to the JS divergence;

[0016] S1 - 3, feature map merging: perform a merging operation on the two - part feature map sets, thus eliminating the problem of too many redundant feature maps in the convolutional layer.

[0017] Further, the S1-1 includes:

[0018] First, the information quantity weight WIQ based on singular value decomposition is used to obtain the correlation between each feature map, calculate the WIQ values corresponding to each feature map in the feature map set G, and sort the feature maps in the feature map set G in descending order according to the WIQ values;

[0019] Secondly, the sorted set Top is used as the top-layer feature map set when merging feature maps. Top is used as the top-layer feature map set containing main features, and the set Bottom, which is a copy of Top, is used as the bottom-layer feature map set when merging feature maps; then, let the feature map X in Top i = WIQ i ·X i + WIQ n-i-1 ·X n-i-1 , and the feature map Y in Bottom i = Y i ·WIQ n-i+1 + Y n-i+1 ·WIQ i , to obtain two reconstructed sets of feature maps;

[0020] Finally, the weighted feature maps in Top and Bottom are respectively divided into upper and lower parts with equal numbers, and the feature maps in these two parts are directly added. The reconstructed feature maps of Top and Bottom finally obtained have a significantly increased effective feature content, and the number of parameters is reduced to half of the original.

[0021] Further, the S1-2 includes:

[0022] First, calculate the mean feature map matrix E corresponding to the set Top;

[0023] Then, the feature correlation coefficient FCC based on JS divergence is used to evaluate the similarity between the current feature map and other feature maps according to the FCC values corresponding to the feature maps in the set Top;

[0024] Finally, calculate the FCC difference between the current feature map and other feature maps. If this value is less than or equal to the threshold η, the current feature map is classified into the redundant feature map set R, otherwise it is retained in the set Top.

[0025] Further, the S1-3 includes:

[0026] First, take one feature map X and Y from the bottom-layer feature map set Bottom and the redundant feature map set R respectively, and calculate the expectation and variance of their corresponding feature matrices;

[0027] Next, the image fusion function MF(X, Y) is used to perform weighted fusion on the redundant feature map Y and the underlying feature map X;

[0028] Finally, use a 1×1 convolution kernel to compress the number of feature maps in the underlying feature map set Bottom to half of the original, and merge the compressed underlying feature map set Bottom and the top-level feature map set Top into a group as the output feature map O.

[0029] Furthermore, the S2 includes:

[0030] S2-1, Parameter initialization: Initialize the various parameters in the model to accelerate the convergence speed of the model parameters;

[0031] S2-2, Parallel CP decomposition: Perform CP decomposition on the convolution kernels in each parallel node, and use the decomposed small convolution kernels to replace the original convolution kernels for convolution operations;

[0032] S2-3, Optimal rank solution: Use an adaptive butterfly optimization algorithm to find the optimal approximate rank combination for CP decomposition.

[0033] Furthermore, the S2-1 includes:

[0034] First, use the weight initialization function KWI(x) to initialize the weight matrices of the convolution kernels of each layer l represents the convolution layer number, and ω represents the total number of convolution layers;

[0035] Then map the feature map and its corresponding convolution kernel into key-value pairs and store them in HDFS;

[0036] Finally, initialize and assign the learning rate μ in the weight update formula, and use the maximum accuracy loss ratio η to represent the accuracy loss brought by CP decomposition;

[0037] Before using CP decomposition for parallel training of the DCNN model, by initializing the convolution kernels of the model and some other variables, the convergence speed of the model training of each node is improved.

[0038] Furthermore, the S2-2 includes:

[0039] S2-2-1, CP decomposition: First, equally divide the input image data X of the current batch l into sub-images and convert these sub-images into file blocks and store them in HDFS; then partition the sub-images through the ParallelCollectionRDD operator and evenly distribute them to each node in the cluster; finally, select the approximate rank ranki Convolution kernels for each node Perform CP decomposition. After supplementing the missing dimensions of the factor matrices obtained from the decomposition, approximate small convolution kernels k1, k2, k3, k4 can be obtained; by performing CP decomposition on the convolution kernel K l Performing CP decomposition on the convolution kernel can reduce the redundant parameters of the convolution kernel;

[0040] S2-2-2, Parallel convolution operation: Perform parallel convolution operations in each partition through the MapPatitionsRDD operator. First, the image blocks to be processed Are successively convolved with the convolution kernels k1, k2, k3, k4 to obtain the corresponding convolution results Then call the shuffledRDD operator to rearrange the intermediate results obtained in each partition For subsequent merging; finally, call the reduceByKey operator to merge the intermediate results obtained in each partition To obtain the output feature map NY l , And store it in memory for use during parameter update;

[0041] S2-2-3, Parameter update: First, calculate the weights of the l-th layer of the current node The corresponding gradient value Then merge the gradient values in each node through the reduceByKey operator To obtain the merged gradient g l ; Finally, update the convolution kernel weights of the (l-1)-th layer through the weight update formula of backpropagation The updated results are all stored in the RDD.

[0042] Furthermore, the S3 includes:

[0043] S3-1, Evaluate data status: Estimate the number of key-value pairs contained in different clusters in the intermediate data through sampling;

[0044] S3-2, Data skew correction: Use the cluster splitting method to evenly distribute the intermediate data into the buckets of each node;

[0045] S3-3, Node parameter merging: Estimate the computing power of each node and allocate the buckets to suitable nodes for data processing.

[0046] Furthermore, the S3-1 includes:

[0047] First, obtain the key-value pairs <key i , value j> Aggregate; then calculate the corresponding data deviation coefficient DCF after passing through the current data partition. When DCF is less than the set threshold, it is considered that the current data partition is relatively reasonable, and the reduceByKey operation can be executed to merge node parameters. Otherwise, it is considered that the intermediate data is severely skewed, and the data needs to be repartitioned again;

[0048] Using the data deviation coefficient DCF to obtain the skewness corresponding to the data volume in each bucket helps to measure the overall deviation of the data partition.

[0049] Further, the S3-2 includes:

[0050] (1) Use the parallel reservoir sampling algorithm to estimate the distribution of the overall data and obtain the data distribution: First, set the capacity of the sample set Sample to R, perform a single scan operation on the key-value pairs to be processed, and add the first r elements scanned to Sample; then replace the elements in S with the remaining k data (k > r) according to the probability r / k; finally, obtain the total amount W of the key-value pairs in the input data and calculate the ratio η between it and the capacity R of the sample set, and then count the number of key-value pairs contained in each data cluster SC j in the sample set, and set C j = SC j × η as the estimated value of each element in the overall data cluster , where C j represents the jth data cluster in the intermediate data, SC j represents the data volume contained in each data cluster in Sample, and m represents the number of overall data clusters;

[0051] (2) After obtaining the estimated value of the data distribution, repartition the data: First, divide the total data volume W by the set number of buckets N as the average data volume W avg that can be accommodated in each bucket, and sequentially set the initial capacity of each bucket B i to W avg ; then sort the data clusters to be allocated in descending order according to the data volume they contain, and sequentially traverse the sorted data clusters C j starting from the first bucket B1. If the data volume n(C j ) corresponding to the data cluster C j is greater than the capacity n(B i ) of the current bucket, then store all the parts of C j that do not exceed the bucket capacity into the bucket, and for the excess part, set the data volume to n(C j ) = n(C j ) - n(B i ) and put it back into the cluster queue as a new cluster. Otherwise, put the data cluster C jAll the data in [[]] is stored in the buckets. At this time, the capacity of the bucket is updated to n(B i ) = n(B i ) - n(C i ), where n(·) represents the data volume corresponding to the data cluster. Finally, the above steps are executed for each bucket in the cluster in turn until all data clusters C j are all stored in the corresponding buckets, and the final partitioning result can be output. By reasonably allocating each cluster to the corresponding bucket, the data skew situation is corrected.

[0052] Furthermore, the S3-3 includes:

[0053] 1) First, it is stipulated that in the first λ - 1 rounds, t data buckets are allocated in sequence, and in the λth round, N - t×(λ - 1) data buckets are allocated. All buckets are sorted in ascending order according to the size of the data volume they can hold. N represents the set number of buckets.

[0054] 2) Second, the node computing power NCP is used to evaluate the data processing capabilities of each node. All nodes are re-sorted in ascending order according to the NCP value. Buckets with less data volume are allocated to Reducers with smaller NCP values (reducers are responsible for merging <key i , value i > with the same key), and buckets with more data volume are allocated to Reducers with larger NCP values.

[0055] 3) Finally, the reduceByKey operator is used to parallelly merge the key-value pairs output by each node, and the merged parameters are used as the final parameter values after training of the DCNN model.

[0056] In summary, due to the adoption of the above technical solutions, the present invention can significantly improve the training efficiency of the DCNN model in a big data environment and at the same time show good scalability on large image data sets. It has the following advantages:

[0057] (1) By using the feature compression strategy FC-SVDJS based on singular value decomposition and JS divergence to group and merge the feature maps in the convolutional layer, the problem of too many redundant feature maps in the training process is solved;

[0058] (2) In combination with Spark, a parallel CP decomposition strategy PCP-ABOA based on adaptive butterfly optimization is proposed to find the optimal approximate convolution kernel to replace the original convolution kernel within a given classification accuracy loss range, so as to reduce the invalid parameters in the convolution kernel and further overcome the problem of low convolution operation efficiency;

[0059] (3) Through the data partitioning strategy DP-CS based on cluster segmentation, the intermediate data is reasonably allocated to the buckets of each node, effectively solving the problem of unbalanced node load.

[0060] Additional aspects and advantages of the present invention will be given in part in the following description, become apparent in part from the following description, or be learned through the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] The above and / or additional aspects and advantages of the present invention will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, wherein:

[0062] Figure 1 is a schematic diagram of the data skew correction process.

[0063] Figure 2 is a schematic diagram of the speedup ratio of the method of the present invention on two data sets.

[0064] Figure 3 is the F-measure value of the method of the present invention on two data sets. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] Embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for explaining the present invention and should not be construed as limiting the present invention.

[0066] For the parallel deep convolutional neural network algorithm in the big data environment, there are problems such as too many redundant feature maps, low efficiency of convolutional operations, and unbalanced node load. The present invention proposes a parallel deep convolutional neural network algorithm PDCNN-SCPBOA based on Spark and CPBOA strategies. First, a feature compression strategy FC-SVDJS based on singular value decomposition and JS divergence is proposed to group and merge the feature maps in the convolutional layer, thus solving the problem of too many redundant feature maps in the training process; then, a parallel CP decomposition strategy PCP-ABOA based on adaptive butterfly optimization is proposed in combination with Spark to find the optimal approximate convolutional kernel to replace the original convolutional kernel within a given classification accuracy loss range, thereby reducing the invalid parameters in the convolutional kernel and overcoming the problem of low efficiency of convolutional operations; finally, a data partitioning strategy DP-CS based on cluster segmentation is proposed to reasonably allocate the intermediate data to the buckets of each node, effectively solving the problem of unbalanced node load. Experiments show that the PDCNN-SCPBOA algorithm can significantly improve the training efficiency of the DCNN model in the big data environment and show good scalability on large data sets.

[0067] 1. Feature Map Compression

[0068] To address the problem of excessive redundant feature maps in parallel DCNN models, the FC - SVDJS strategy is proposed. This strategy mainly consists of three parts: (1) Feature map reconstruction: The feature map is reconstructed into two parts with more effective information through singular value decomposition; (2) Redundant feature map filtering: Redundant feature maps in the feature map are filtered according to the JS divergence; (3) Feature map merging: A merging operation is performed on the two sets of feature maps, thus eliminating the problem of excessive redundant feature maps in the convolutional layer.

[0069] (1) Feature map reconstruction

[0070] To eliminate redundant feature maps in the convolutional layer, it is necessary to first re - sort the input feature map set G, and then perform a grouping and merging operation on the new feature map set, so as to achieve the effect of initially eliminating redundant feature maps. The specific steps are as follows: First, an information quantity weight WIQ based on singular value decomposition is proposed to obtain the correlation between each feature map, calculate the WIQ values corresponding to each feature map in the feature map set G, and sort the feature maps in the feature map set G in descending order according to the size of the WIQ values; Second, copy the sorted set into two identical feature map sets Top and Bottom. Top is used as the top - layer feature map set containing the main features, and Bottom is used as the bottom - layer feature map set to supplement the missing features; Then, multiply each feature map X i in Top by its corresponding WIQ value to obtain the forward - weighted feature map, and multiply each feature map X i in Bottom by the WIQ value of the feature map X n-i+1 at the symmetric position to obtain the reverse - weighted feature map; Finally, divide the weighted feature maps in Top and Bottom into two equal upper and lower parts respectively, and directly add the feature maps in these two parts. The effective feature content in the two sets of reconstructed feature maps obtained finally is greatly improved, and the number of parameters is reduced to half of the original.

[0071] Theorem 1 (Information Quantity Weight WIQ) Assume that the matrix corresponding to the i - th feature map in the current convolutional layer is X i , and its rank is r, then the information quantity weight WIQ corresponding to this feature map is:

[0072]

[0073] w i =Rank(X' i ) (2)

[0074]

[0075] where, w iDenote the rank corresponding to the i-th low-rank matrix, and ||·||1 represents the l1 norm;

[0076] Rank() represents obtaining the rank of the low-rank matrix obtained after performing SVD decomposition on X i ' of the low-rank matrix obtained after performing SVD decomposition, and u i , is the orthogonal matrix U m×m and the i-th eigenvector of V n×n , and σ i is the i-th diagonal element of the diagonal matrix ∑ m×n .

[0077] Proof: Given the eigenmap matrix X i After performing SVD decomposition, a low-rank matrix X i ' and a matrix containing redundant information can be obtained. w i Denotes the rank corresponding to the low-rank matrix, and this value reflects the amount of information carried by the eigenmap. When w i has a large value, it indicates that the current eigenmap contains more information and should be retained; when w i has a small value, it indicates that the current eigenmap contains less information and should be eliminated. WIQ represents the value after normalizing w i , and this value reflects the proportion of the information contained in the current eigenmap in the reconstructed eigenmap. When the value of WIQ is closer to 1, it indicates that the proportion of the information contained in the current eigenmap in the reconstructed eigenmap is larger; when the value of WIQ is closer to 0, it indicates that the proportion of the information contained in the current eigenmap in the reconstructed eigenmap is smaller. Q.E.D.

[0078] In summary, by comparing the values of WIQ, the eigenmaps can be reordered according to the information content, which is more conducive to obtaining the reconstructed eigenmap.

[0079] (2) Redundant eigenmap filtering

[0080] After performing feature fusion on the eigenmaps in the top-level eigenmap set Top during the feature reconstruction stage, there are still some redundant eigenmaps in the generated reconstructed eigenmap set. Therefore, further feature filtering operations need to be performed on this set. The specific process is as follows: First, calculate the mean eigenmap matrix E corresponding to the set Top; then, propose the feature correlation coefficient FCC based on the JS divergence, and evaluate the similarity between the eigenmaps in the set Top according to the FCC values corresponding to each eigenmap; finally, calculate the FCC difference between the current eigenmap and other eigenmaps. If this value is less than or equal to the threshold η, the current eigenmap is classified into the redundant eigenmap set R, otherwise it is retained in the set Top.

[0081] Theorem 2 (Feature Correlation Coefficient FCC) Assume that the total number of input eigenmaps of the convolutional layer is n, and the i-th eigenmap is Xi , the mean feature map corresponding to all feature maps is E, and X i The vector representation forms of and Then the feature map X i The corresponding feature correlation coefficient FCC can be expressed as:

[0082] FCC i = JS i ·dist(X i , E) (4)

[0083] Among them,

[0084]

[0085]

[0086] JS i represents the JS divergence value between the i-th feature map X i and the mean feature map E;

[0087] dist(X i , E) represents the degree of deviation between X i and E;

[0088] represents the KL divergence between X i and , also known as relative entropy.

[0089] ||·||2 represents the second norm;

[0090] Proof: Given that the mean feature map E represents the overall feature distribution in the feature map set G, the similarity between feature maps can be measured by calculating the degree of association between different feature maps and E. JS i represents the JS divergence value between the i-th feature map X i and the mean feature map E, and this value reflects the correlation degree between X i and E, and its value range is from 0 to 1. When the value of JS i is closer to 0, it indicates that the correlation degree between X i and E is greater, and the two can be substituted for each other; when the value of JS i is closer to 1, it indicates that the association degree between X i and E is smaller, and the two cannot be substituted for each other. dist(X i , E) represents the degree of deviation between X i and E, and the larger its value, the greater the feature deviation degree between X i and E, and vice versa. In summary, when JS iand the product between dist(X i , E) is large, it indicates that the current feature map X i has a greater degree of correlation with the mean feature map; when the product between JS i and dist(X i , E) is small, it indicates that the current feature map X i has a smaller degree of correlation with the mean feature map. Q.E.D.

[0091] Therefore, by calculating the difference in the feature correlation coefficients FCC corresponding to different feature maps, their degree of correlation can be evaluated. Feature maps with smaller differences should be classified into the redundant feature set, and those with larger differences should be retained in the original feature map set.

[0092] (3) Feature map merging

[0093] After feature filtering, the bottom feature map set Bottom needs to be feature compressed before merging with the top feature map set Top to obtain the final output feature map set. The specific process is as follows: First, take a feature map X and Y from the bottom feature map set Bottom and the redundant feature map set R respectively, and calculate the expectation and variance of their corresponding feature matrices; then, propose an image fusion function MF(X, Y) to perform weighted fusion of the redundant feature map Y and the bottom feature map X; finally, use a 1×1 convolution kernel to compress the number of feature maps in the bottom feature map set Bottom to half of the original, and merge the compressed bottom feature map set Bottom and the top feature map set Top as a group to serve as the output feature map O.

[0094] Theorem 3 (Image fusion function MF(X, Y)) Assume that X and Y are the matrix forms of two feature maps to be fused, μ x and μ y are the expectations of matrices X and Y respectively, σ x and σ y are the variances of matrices X and Y respectively, then the expression of the image fusion function MF(X, Y) is:

[0095] MF(X, Y) = X + λ·Y (7)

[0096] Among them,

[0097]

[0098] cov(X, Y) = E[(X - μ x )(Y - μ y )] (9)

[0099] λ·Y represents extracting redundant features using the similarity coefficient as the weight;

[0100] cov(X, Y) represents the degree of correlation between vectors X and Y;

[0101] |·| is the absolute value symbol;

[0102] σ x and σ y are the variances of vectors X and Y respectively;

[0103] E represents taking the expectation;

[0104] X and Y are the matrix forms of two feature maps to be fused respectively;

[0105] μ x and μ y are the expected values of feature matrices X and Y respectively.

[0106] Proof: Given that μ x and μ y are the expected values of feature matrices X and Y respectively, the covariance cov(X, Y) obtained through μ x and μ y represents the degree of correlation between vectors X and Y. When the value of |cov(X, Y)| is closer to 0, it indicates that the similarity between feature matrices X and Y is smaller, and vice versa. Also known that σ x and σ y are the variances of vectors X and Y respectively. Dividing the absolute value of cov(X, Y) by σ x σ y can obtain the similarity coefficient λ, which eliminates the influence of dimension and narrows the range of cov(X, Y) to between 0 and 1. When λ = 1, the similarity between feature matrices X and Y is the largest; when λ = 0, the similarity between feature matrices X and Y is the smallest. λ·Y represents extracting redundant features using the similarity coefficient as the weight, and adding this weighted matrix to the feature map matrix X can obtain the corresponding fused feature matrix. Q.E.D.

[0107] In summary, the image fusion function MF(X, Y) can effectively fuse two feature maps, and the new underlying feature map after fusion contains more effective features.

[0108] The pseudo-code of the FC-SVDJS strategy is shown in Algorithm 1:

[0109] Algorithm 1 Feature Map Compression

[0110]

[0111]

[0112]

[0113] 2. Model Parallel Training

[0114] To address the problem of low efficiency in convolutional operations during the parallel training of DCNN models, the PCP-ABOA strategy is proposed. By performing CP decomposition on the convolutional kernels in each node's model, the convolutional operations are accelerated. This strategy mainly consists of three parts: (1) Parameter initialization: Initialize the various parameters in the model to accelerate the convergence speed of the model parameters; (2) Parallel CP decomposition: Perform CP decomposition on the convolutional kernels in each parallel node, and use the decomposed small convolutional kernels to replace the original convolutional kernels for convolutional operations; (3) Optimal rank solution: Since the error caused by the initial rank of CP decomposition is relatively large, the adaptive Butterfly Optimization Algorithm (BOA) is used to find the optimal approximate rank combination for CP decomposition. Among them, CP decomposition mainly decomposes the convolutional kernel tensor by selecting different ranks, and this decomposition will cause a loss of classification accuracy. Using an intelligent optimization algorithm to find the optimal rank for CP decomposition can minimize the loss of classification accuracy.

[0115] The specific operation steps of each module are as follows:

[0116] (1) Parameter initialization

[0117] Before using CP decomposition for parallel training of the DCNN model, to improve the convergence speed of each node's model training, it is necessary to initialize the convolutional kernels and some other variables in the model. The specific process is as follows: First, a weight initialization function KWI(x) is proposed to initialize the weight matrices of the convolutional kernels of each layer Then, the feature map and its corresponding convolutional kernel are mapped into key-value pairs and stored in HDFS; finally, the learning rate μ in the weight update formula is initialized and assigned, and the accuracy maximum loss ratio η is used to represent the accuracy loss caused by CP decomposition.

[0118] Theorem 4 (Weight Initialization Function KWI(x)) Assume that the number of convolutional layers in the training model is n, the number of input feature maps in the l-th layer is cl, and the feature matrix of each image is The weight matrix of the convolutional kernel is with a size of r×r, then the weight initialization function KWI(x) is:

[0119]

[0120] where,

[0121]

[0122]

[0123] E iThe initialization matrix representing the convolutional kernel;

[0124] Denote ξ1, ξ2,..., ξ r Subject to a normal distribution with a mathematical expectation of 0 and a variance of ;

[0125] cl represents the number of input feature maps in the l-th layer;

[0126] r 2 Represents the size of the convolutional kernel weight matrix ;

[0127] Proof: Given that the feature matrix of the input image in the l-th layer is The number is cl, and the weight matrix of the convolutional kernel is With a size of r×r, if the ReLu function is used as the activation function f(x) for forward propagation, then the input of the l-th layer is Denote the bias, and its corresponding expected value and variance are:

[0128]

[0129]

[0130] Since each element x In the matrix l Are independent of each other and follow the same distribution ( Each element w l Also has this characteristic), then the above calculation formulas for the expected value and variance can be changed to:

[0131] E(x l ) = c l ×r 2 ×E(w l )E(f(x l-1 )) (15)

[0132] Var(x l ) = c l ×r 2 ×Var(w l f(x l-1 )) (16)

[0133] Where, E(w l ) and E(f(x l-1 )) respectively represent the variance values of the distributions followed by the elements in the matrices And ;

[0134] For the expected value E(w l), if its value is not 0, it means that the parameter values of the model will diverge during the forward propagation process, which is very likely to lead to the occurrence of gradient explosion. Therefore, it must satisfy E(w l ) = 0. At this time, it can be obtained that E(f(x l-1 )) = 0. Then, according to the relationship formula between variance and expectation, we have:

[0135] Var(w l f(x l-1 )) = E([w l 2 )E([f(x l-1 )] 2 ) = Var(w l )E([f(x l-1 )] 2 ) (17)

[0136] Then formula (29) can be changed to:

[0137] Var(x l ) = c l ×r 2 ×Var(w l )E([f(x l-1 )] 2 ) (18)

[0138] Since the activation function f(x l-1 ) is the ReLu function, its value has a probability of 0 for half of the cases and a probability of x l-1 for the other half. Then it can be obtained that:

[0139]

[0140] To prevent the model from overfitting, the variances corresponding to x l and x l-1 should be kept consistent. Then substituting formula (32) into formula (31) gives:

[0141]

[0142] When the parameters w l of each layer of convolutional kernels are initialized with a normal distribution, the model training is not likely to have adverse phenomena such as gradient explosion and overfitting. Then it can be known that Q.E.D.

[0143] In summary, using the weight initialization function KWI(x) can effectively initialize the convolutional kernels and improve the training speed of the model.

[0144] (2) Parallel CP decomposition

[0145] ​After initializing the various parameters of the DCNN model, the parallel CP decomposition of the convolution kernels of each layer of the model can be carried out in combination with the Spark framework. The specific process is as follows:

[0146] CP decomposition: By performing CP decomposition on the convolution kernel K l to reduce the redundant parameters of the convolution kernel. First, the input image data X of the current batch l is cut into sub-images of equal size and these sub-images are converted into file blocks and stored in HDFS; then, the sub-images are partitioned through the ParallelCollectionRDD operator and evenly distributed to each node of the cluster; finally, an approximate rank rank i is selected to perform CP decomposition on the convolution kernels of each node . After supplementing the missing dimensions of the factor matrices obtained by the decomposition, approximate small convolution kernels k1, k2, k3, k4 can be obtained.

[0147] Parallel convolution operation: Perform parallel convolution operations in each partition through the MapPatitionsRDD operator. First, the image blocks to be processed are successively convolved with the convolution kernels k1, k2, k3, k4 to obtain the corresponding convolution results Then, the shuffledRDD operator is called to rearrange the intermediate results Y obtained in each partition i l for subsequent merging; finally, the reduceByKey operator is called to merge the intermediate results Y obtained in each partition i l to obtain the output feature map NY l , which is then stored in memory for use during parameter update.

[0148] Parameter update: After the parallel convolution operation, the parameters of each layer of the model can be updated. First, calculate the gradient value corresponding to the weight of the l-th layer of the current node Then, the gradient values in each node are merged through the reduceByKey operator to obtain the merged gradient g ; finally, the convolution kernel weights of the (l - 1)-th layer are updated through the weight update formula of backpropagation l and the updated results are all stored in the RDD. (3) Optimal rank solution

[0149] (3) Optimal rank solution

[0150] In the parallel CP decomposition process, the approximately selected ranks will cause a large accuracy loss in the training results. Therefore, an adaptive butterfly optimization algorithm is proposed to optimize the approximately selected ranks of CP decomposition. The specific steps are as follows:

[0151] 1) Initialize the population: Take the initial approximate ranks {rank1, rank2,..., rank n} of each convolutional layer as the initial butterfly population. Set the maximum number of iterations of the population to T = 1000. Calculate the dynamic switching probability p of the t-th generation population using formula (38), and use the accuracy loss ratio η as the fitness to screen the butterfly populations with different rank combinations.

[0152] p = 0.6 - 0.1×(T - t) / T (21)

[0153] 2) Calculate the fragrance intensity: When calculating the fragrance intensity of butterfly individuals using the formula f = c t I a it often leads to poor convergence speed and search effect of the BOA algorithm. Therefore, an adaptive fragrance coefficient AFC is proposed to replace the original fragrance concentration f, so as to improve the convergence speed of the butterfly optimization algorithm. Then calculate the AFC value corresponding to each butterfly individual, and assign the maximum AFC value to Assign the minimum AFC value to c t represents the perception factor of the t-th generation, and I a represents the stimulation intensity.

[0154] 3) Update the position of butterfly individuals: Randomly initialize the value of r in the range of 0 to 1. At the same time, to prevent the BOA algorithm from falling into a local optimal solution, a position update function PU(x) is proposed to enable the BOA algorithm to maintain good search ability. If r < p, perform global search to update the position, otherwise perform local search to update the position.

[0155] 4) Update the population: Repeat steps 2 and 3 until the position distribution of the current butterfly population can make the loss ratio η of the classification accuracy reach the minimum threshold, that is, the corresponding approximate rank combination {rank'1, rank'2,..., rank' n} in each convolutional layer can make the model acceleration effect reach the optimal.

[0156] Theorem 6 (Adaptive fragrance coefficient AFC): Assume that the number of iterations of the current butterfly population is t and the maximum number of iterations is T. Then the adaptive fragrance coefficient AFC can be expressed as:

[0157] AFC = u×(1 - σ) (22)

[0158]

[0159] Among them, u is a constant that follows the standard normal distribution N(0, 1), σ represents the standard deviation, and m is a tuning parameter used to scale the ratio between t and T.

[0160] Proof: Since the fragrance coefficient f = c in the traditional BOA algorithm t I a has randomness and blindness, an adaptive algorithm is selected to automatically adjust the distribution range of the fragrance. u follows the standard normal distribution, which can make the distribution of the fragrance coefficient more in line with the natural state. When the ratio between the iteration number t of the butterfly population and the maximum iteration number T is small, the larger the value of 1 - σ, it indicates that the distribution range of the fragrance should be increased at this time to make the algorithm search more carefully; when the ratio between the iteration number t of the butterfly population and the maximum iteration number T is large, the smaller the value of 1 - σ, it indicates that the distribution range of the fragrance should be reduced at this time to search for the optimal value more quickly. In summary, the product of u and 1 - σ can automatically adjust the distribution of the fragrance to meet the needs of optimization. Q.E.D.

[0161] The adaptive fragrance coefficient AFC can be used to improve the convergence speed of the BOA algorithm and the ability to jump out of local extreme points. However, the position update function of the traditional BOA algorithm still brings the problem of local optimal values. Therefore, this paper proposes an improved position update function PU(x).

[0162] Theorem 7 (Position update function PU(x)): Assume is the position of the i-th butterfly individual in the t-th generation population, is the position of the i-th butterfly individual in the (t + 1)-th generation population, is the position of the butterfly individual with the largest fragrance coefficient in the t-th generation population, is the position of the butterfly individual with the smallest fragrance coefficient in the t-th generation population, r is a random number in the range of 0 to 1, p is the switching probability, and AFC is the fragrance coefficient of the butterfly individual Then the position update function PU(x) is:

[0163]

[0164] Proof: In the global search formula , represents the position of the optimal butterfly individual. By adjusting the difference between the optimal butterfly and each butterfly individual in the current population, the butterfly individuals can be distributed around the optimal butterfly to wander, so as to conduct a more thorough search. In the local search formula , the position of the next generation of butterfly population is updated by the average value of the optimal butterfly position and the worst butterfly position in each iteration, which can reduce the blindness when randomly selecting butterfly individuals to follow. Q.E.D.

[0165] Therefore, the position update function PU(x) can be used as the position update function for butterfly individuals during the search of the BOA algorithm to reduce the blindness of butterfly position updates, thereby avoiding the optimal solution output from falling into a local optimum.

[0166] The pseudo-code of the PCP-ABOA strategy is shown in Algorithm 2:

[0167] Algorithm 2 Model Parallel Training

[0168]

[0169]

[0170]

[0171] 3. Parameter Parallel Merging

[0172] Aiming at the problem of data skew among cluster nodes that is prone to occur during the parameter parallel merging process, the DP-CS strategy is proposed to evenly distribute various intermediate data generated during the model parallel training process, thereby avoiding data skew. Specifically, it includes: evaluating the data status: estimating the number of key-value pairs contained in different clusters in the intermediate data through sampling; data skew correction: using the cluster splitting method to evenly distribute the intermediate data into the buckets of each node; node parameter merging: estimating the computing power of each node and allocating the buckets to suitable nodes for data processing.

[0173] (1) Evaluating the data status

[0174] The intermediate data during the Spark operation always exists in the form of <key, value> key-value pairs, and these key-value pairs are generally stored in the buckets created by the Map operation. Since the Hash partitioning method will store the intermediate data with the same key value in the same bucket, when there is too much intermediate data with the same key value, it often leads to data skew, which seriously affects the efficiency of the final parameter merging. Therefore, the data deviation coefficient DCF is proposed to obtain the skew degree corresponding to the data volume in each bucket, so as to measure the overall deviation of data partitioning. The specific process is as follows: First, obtain the set of key-value pairs <key i , value j > contained in the intermediate output data of the previous stage; then calculate the corresponding data deviation coefficient DCF after the current data partitioning. When DCF is less than the set threshold, it is considered that the current data partitioning is relatively reasonable, and the reduceByKey operation can be executed to merge node parameters. Otherwise, it is considered that the intermediate data is severely skewed and the data needs to be repartitioned.

[0175] Theorem 8 (Data Deviation Coefficient DCF): Assume that there are N buckets in the Spark cluster. When storing intermediate data, it is queried that the j-th bucket contains l data clusters, and the size of the k-th data cluster is Then the data deviation coefficient DCF can be expressed as:

[0176]

[0177]

[0178]

[0179] where, I j and I avg respectively represent the total size of the space occupied by the j-th bucket and the average space size occupied by each bucket. S(I j ) and S avg respectively represent the total size of the space occupied by the j-th bucket after the shuffle operation and the average space size occupied by each bucket.

[0180] Proof: Before and after the shuffle operation, there will be a certain difference in the number of tuples in the data cluster, resulting in the traditional coefficient of variation being unable to accurately evaluate the data distribution. Therefore, it is necessary to take S(I j ) and S avg as one of the factors for calculating the data skew, that is, take the weighted average of I j and S(I j ) as the sample element value in formula (42) to calculate the corresponding standard deviation σ and mean μ:

[0181]

[0182] μ = 0.5 · (I avg + S avg ) (29)

[0183] The ratio of the above two formulas is the coefficient of variation DCF corresponding to the intermediate data:

[0184]

[0185] This value is used to represent the degree of dispersion of the data distribution in data statistics. When the DCF value is smaller, the amount of data stored in each bucket is closer, indicating that the data skew in the bucket is smaller at this time; when the DCF value is larger, the difference in the amount of data stored in each bucket is larger, indicating that the data skew in the bucket is larger at this time. Q.E.D.

[0186] Therefore, the coefficient of variation of data CVD can be used to calculate the data skew of the intermediate data in order to evaluate the result of the final data partitioning.

[0187] (2) Data skew correction

[0188] When data skew occurs in a node, in order to prevent the merging efficiency of parameters from being affected, it is necessary to re - allocate the intermediate data. Therefore, a data partitioning method based on cluster segmentation is proposed to re - partition key - value pairs. The specific process is as Figure 1 follows:

[0189] (1) Obtain data distribution: Statistical analysis of the entire data set consumes a large amount of resources. Therefore, a parallel reservoir sampling algorithm is used to estimate the distribution of the overall data. The specific process is as follows: First, set the capacity of the sample set Sample to R, perform a single - scan operation on the key - value pairs to be processed, and add the first r elements scanned to Sample; then replace the elements in S with the remaining k data (k > r) according to the probability r / k; finally, obtain the total amount W of key - value pairs in the input data and calculate the ratio η between it and the capacity R of the sample set. Then count the number of key - value pairs contained in each data cluster SC j in the sample set, and set C j = SC j ×η as the estimated value of each element in the overall data cluster . C j represents the j - th data cluster in the intermediate data, SC j represents the j - th data cluster in the sample, and m represents the number of overall data clusters.

[0190] (2) Data re - partitioning: After obtaining the estimated value of the data distribution, it is necessary to reasonably allocate each cluster to the corresponding bucket to correct the data skew situation. The specific process is as follows: First, divide the total data amount W by the set number of buckets N to obtain the average data amount W avg that can be accommodated in each bucket. Then, set the initial capacity of each bucket B i to W avg ; then sort the data clusters to be allocated in descending order according to the amount of data they contain. Starting from the first bucket B1, sequentially traverse the sorted data clusters C j . If the amount of data n(C j ) corresponding to the data cluster C j is greater than the capacity n(B i ) of the current bucket, then store all the part of C j that does not exceed the bucket capacity into the bucket, and for the exceeded part, set the amount of data to n(C j ) = n(C j ) - n(B i ) and re - put it into the cluster queue as a new cluster. Otherwise, put the data cluster C jAll the data in it is stored in the buckets. At this time, the capacity of the bucket is updated to n(B i ) = n(B i ) - n(C i ); Finally, the above steps are executed for each bucket in the cluster in turn until each data cluster C j is all stored in the corresponding bucket, and then the final partitioning result can be output.

[0191] (3) Node parameter merging

[0192] After performing data skew correction, each bucket needs to be mapped to the corresponding Reducer. Since the number of both is equal, the system will establish a mapping relationship between them by default, ignoring the differences in CPU computing power and memory size. Therefore, a bucket allocation method based on node computing power is proposed to effectively balance the relationship between the amount of tasks to be processed and the node computing power, thereby accelerating the process of node parameter merging. The specific process is as follows:

[0193] 1) First, it is stipulated that in the first λ - 1 rounds, t data buckets are allocated in turn, and in the λ-th round, N - t×(λ - 1) data buckets are allocated. All the buckets are sorted in ascending order according to the amount of data they can hold;

[0194] 2) Secondly, a node computing power NCP is proposed to evaluate the data processing ability of each node. All the nodes are re-sorted in ascending order according to the NCP value. Buckets with less data capacity are allocated to Reducers with smaller NCP values, and buckets with more data capacity are allocated to Reducers with larger NCP values;

[0195] 3) Finally, the reduceByKey operator is used to parallelly merge the key-value pairs output by each node, and the merged parameters are used as the final parameter values of the DCNN model after training.

[0196] Theorem 9 (Node computing power NCP): Assume that the number of nodes in the parallel framework is set to n, the main frequency of the CPU corresponding to each node is f i , the number of cores is C i , the memory space is S i , the CPU occupancy rate obtained through system performance monitoring is μ i , and the memory resource occupancy rate is ν i , where i ∈ (0, n). Then the expression of the node computing power NCP is:

[0197] NCP i = (1 - μ i )·CPU i + (1 - ν i )·MS i (31)

[0198] Among them,

[0199]

[0200]

[0201] μ i represents the CPU occupancy rate of the i-th node;

[0202] CPU i represents the computing power of the i-th node after normalization;

[0203] MS i represents the size of the storage space allocated to the i-th node after normalization;

[0204] S i represents the memory space of the i-th node;

[0205] S max and S min represent the maximum memory space and the minimum memory space respectively.

[0206] Proof: It is known that the node computing resources of the Spark framework mainly include two parts: CPU resources and memory resources. When the cluster nodes are processing tasks, their processing capabilities are mainly reflected in the amount of data processed per unit time. When calling the CPU to execute tasks, the product of the main frequency and the number of cores f i ×C i is used to represent the number of tasks processed by the CPU per unit time; when calling the memory to execute tasks, S i is used to represent the size of the memory space allocated to the node. 1 - μ i represents the ratio of the node resources that can still be used in the system. (1 - μ i )·CPU i represents the actual computing power of the current CPU. The larger this value is, the stronger the computing power of the current node; conversely, it is weaker. (1 - ν i ) represents the ratio of the memory resources that can still be allocated in the system. (1 - ν i )·MS i represents the available memory space of the current node. The larger this value is, the more data the current node can process; conversely, it is less. The sum of the two gives the NCP value, which comprehensively reflects the computing power of the current node. Q.E.D.

[0207] Therefore, the calculated NCP value can be used to represent the node computing power. By comparing the NCP values of different nodes, the corresponding nodes can be reasonably stored in the Reducer.

[0208] The pseudo-code of the DP-CS strategy is shown in Algorithm 3:

[0209] Algorithm 3 Parameter Parallel Merge

[0210]

[0211]

[0212] 4. Validation of the effectiveness of the PDCNN-SCPBOA algorithm

[0213] To verify the effectiveness and feasibility of the PDCNN-SCPBOA algorithm, Yan'an City and Ansai District of Yan'an City were rasterized, obtaining 60,185,000 and 5,600,000 grid cells respectively. Each grid cell has 7 influencing factors (dimensions), and they are respectively called the Yan'an City dataset and the Ansai District of Yan'an City dataset. The PDCNN-SCPBOA algorithm was applied to the above two datasets for performance verification of the algorithm.

[0214] 4.1 Parallelism analysis of the RBFO-PSVM method

[0215] To verify the feasibility of the PDCNN-SCPBOA algorithm for training DCNNs in a big data environment, the speedup was used as the evaluation index, and experiments were conducted on these two datasets of Yan'an City and Ansai District of Yan'an City. To further ensure the accuracy of the experiment, the speedup was calculated based on the average result after running 10 times. The experimental results are as Figure 2 shown.

[0216] From Figure 2It can be seen that the speedup ratios of the algorithm on the Yan'an City dataset and the Ansai District dataset of Yan'an City both gradually increase with the increase in the number of nodes. The increase in the speedup ratio when the algorithm runs on two nodes is relatively small because the amount of data allocated to each node is large, and its parallel performance has not been significantly improved. As the number of nodes increases, the speedup ratio shows a linear increase. When the number of nodes increases to 6, when the algorithm processes the Yan'an City dataset, the speedup ratio reaches 4.47, which is 3.47 higher than that of a single node; when processing the Ansai District dataset of Yan'an City, the speedup ratio is as high as 3.96, which is 2.96 higher than that of a single node. The main reasons for this result include: (1) When the number of nodes is small, the overall running time of the algorithm is mainly determined by the communication time between nodes, and the operation time reduced by parallel computing is limited and cannot play a dominant role, resulting in a small speedup ratio of the algorithm at this time; (2) When the number of nodes is large, the PDCNN-SCPBOA algorithm effectively eliminates the redundant parameters in the feature map and the convolution kernel matrix by virtue of the feature compression strategy FC-SVDJS and the parallel convolution decomposition strategy PCP-ABOA, improves the convolution operation efficiency, and reduces the model training time; (3) When parallelly merging the model parameters, the PDCNN-SCPBOA algorithm designs the DP-CS strategy, effectively partitions the data, and prevents the node load imbalance caused by data skew of the intermediate data, thereby improving the parallel operation efficiency of each node. Therefore, the performance improvement of the PDCNN-SCPBOA algorithm on the parallel computing platform will be more significant with the increase in the number of nodes, which also indicates that the PDCNN-SCPBOA algorithm is suitable for the DCNN model training in the big data environment and has good feasibility.

[0217] 4.2 Classification Accuracy Analysis of the PDCNN-SCPBOA Method

[0218] To analyze the classification accuracy of the PDCNN-SCPBOA algorithm, experiments were conducted on two datasets in Yan'an City and Ansai District of Yan'an City, and the results of the accuracy F-measure of the algorithm are as Figure 3 shown.

[0219] From Figure 3It can be seen that the classification accuracy of the PDCNN-SCPBOA algorithm on both datasets has reached over 85%. On the Yan'an City dataset, the F-measure value of the algorithm reached 88.7%; on the Ansai District dataset of Yan'an City, the F-measure value of the algorithm reached 86.6%. The main reasons for this result are as follows: (1) The PDCNN-SCPBOA algorithm designs the FC-SVDJS strategy for feature compression, which improves the classification ability of the DCNN model by enriching the information content of the feature map and filtering redundant features; (2) The PDCNN-SCPBOA algorithm designs the PCP-ABOA strategy for iteratively solving the optimal rank, which automatically searches for the optimal approximate rank of the CP decomposition within a given precision loss interval, greatly reducing the error caused by manual rank selection, thereby improving the classification accuracy of the model. Therefore, from the above comparison results, it can be seen that the PDCNN-SCPBOA algorithm has a more excellent classification accuracy on the four datasets.

[0220] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A parallel DCNN classification method for large - area landslide hazard assessment, characterized in that, It includes the following steps: S1. Perform grouped merging on the feature maps in the convolutional layer to achieve feature map compression. S1 includes: S1-1, Feature map reconstruction: Reorder the input feature map set and reconstruct the feature map into two sets of feature maps through singular value decomposition; S1-2. Redundant feature map filtering: Filter out the redundant feature maps in the feature maps according to the Jensen-Shannon divergence. S1-3. Feature map merging: Perform a merging operation on the two parts of the feature map sets. S2. During the parallel training of the model, find the optimal approximate convolutional kernel within the given classification accuracy loss range to replace the original convolutional kernel. S2 includes: S2-1. Parameter initialization: Initialize the various parameters in the model to accelerate the convergence speed of the model parameters. S2-2. Parallel CP decomposition: Perform CP decomposition on the convolutional kernels in each parallel node, and use the decomposed small convolutional kernels to replace the original convolutional kernels for convolution operations. S2-3. Optimal rank solution: Find the optimal approximate rank combination of CP decomposition through an adaptive butterfly optimization algorithm. S3. Uniformly distribute various intermediate data generated during the parallel training of the model. S3 includes: S3-1. Evaluate the data status: Estimate the number of key-value pairs contained in different clusters in the intermediate data by sampling. S3-2. Data skew correction: Use the cluster splitting method to evenly distribute the intermediate data into the buckets of each node. S3-3. Node parameter merging: Estimate the computing power of each node and allocate the buckets to the appropriate nodes for data processing.

2. The parallel DCNN classification method for large-area landslide hazard assessment according to claim 1, characterized in that S1-1 includes: First, the information quantity weight based on singular value decomposition is adopted to obtain the correlation between each feature map, and calculate the feature map set corresponding to each feature map in the numerical value, and sort the feature maps in the feature map set in descending order according to the size of the numerical value; ​ Secondly, the sorted set is used as the top-level feature map set when merging the feature maps, is used as the top-level feature map set containing the main features, the copied set is used as the bottom-level feature map set when merging the feature maps; then, let the feature maps in , the feature maps in to obtain two reconstructed sets of feature maps; Finally, the weighted feature maps in and are respectively divided into upper and lower parts with equal quantities, and the feature maps of these two parts are directly added together. Finally, the reconstructed feature maps of and are obtained.

3. The parallel DCNN classification method for large-area landslide hazard assessment according to claim 1, characterized in that S1-2 includes: First, calculate the set corresponding mean feature map matrix ; Then, the feature correlation coefficient based on JS divergence is adopted , and according to the set corresponding to each feature map in the numerical value to evaluate the similarity between the current feature map and other feature maps; Finally, calculate the difference between the current feature map and other feature maps. If this value is less than or equal to the threshold then classify the current feature map into the redundant feature map set otherwise, retain it in the set .

4. The parallel DCNN classification method for large-area landslide hazard assessment according to claim 1, characterized in that S1-3 includes: First, take one feature map from the set of underlying feature maps and the set of redundant feature maps respectively, and calculate the expectations and variances of their corresponding feature matrices and ; Next, an image fusion function is used to perform weighted fusion on the redundant feature map and the underlying feature map ; Finally, use the convolutional kernel to compress the number of feature maps in the set of underlying feature maps to half of the original, and combine the compressed set of underlying feature maps with the set of top-level feature maps into a group as the output feature maps .

5. The parallel DCNN classification method for large-area landslide hazard assessment according to claim 1, characterized in that, S2-1 includes: First, use the weight initialization function to initialize the weight matrices of the convolutional kernels of each layer , represents the convolutional layer number, represents the total number of convolutional layers; Then map the feature map and its corresponding convolution kernel into key-value pairs and store them in HDFS; Finally, the learning rate in the weight update formula is initialized and assigned, and the maximum loss ratio of accuracy is used to represent the precision loss caused by CP decomposition. ​ 6. The parallel DCNN classification method for large-area landslide hazard assessment according to claim 1, wherein, S2-2 includes: S2-2-1, CP Decomposition: First, cut the input image data of the current batch into sub-images of the same size , and convert these sub-images into file blocks and store them in HDFS; then partition the sub-images through the ParallelCollectionRDD operator and distribute them evenly to each node of the cluster; finally, select the approximate rank to perform CP decomposition on the convolutional kernels of each node . After supplementing the missing dimensions of the factor matrices obtained from the decomposition, approximate small convolutional kernels can be obtained; S2-2-2, Parallel Convolution Operation: Execute parallel convolution operations in each partition through the MapPatitionsRDD operator. First, the image blocks to be processed are successively convolved with the convolution kernel to obtain the corresponding convolution results ; then call the shuffledRDD operator to rearrange the intermediate results obtained in each partition for subsequent merging; finally, call the reduceByKey operator to merge the intermediate results obtained in each partition to obtain the output feature map , and store it in memory for use during parameter update; S2-2-3, Parameter Update: First, calculate the weights of the layer of the current node corresponding gradient values ; then use the reduceByKey operator to merge the gradient values in each node to obtain the merged gradient ; finally, use the weight update formula of backpropagation to update the convolution kernel weights of the layer, and store all the updated results in the RDD.

7. The parallel DCNN classification method for large-area landslide hazard assessment according to claim 1, characterized in that, S3-1 includes: First, obtain the key-value pairs contained in the intermediate output data during the model parallel training phase set; then calculate the corresponding data deviation coefficient after the current data partition , when is less than the set threshold, it is considered that the current data partition is reasonable, and the reduceByKey operation is executed to merge the node parameters. Otherwise, it is considered that the intermediate data is severely skewed and the data needs to be repartitioned

Images