A parallel K-means clustering method based on MapReduce and MSSA

Abstract

This paper proposes a parallel K-means clustering method based on MapReduce and MSSA, comprising the following steps: S1, performing feature weighting and feature extraction on high-dimensional data; S2, partitioning the feature-extracted data, selecting high-quality partitioning points through two-segment mapping, and performing generalized hyperplane partitioning on the data based on the partitioning points to obtain uniform partitions on the map; S3, optimizing the centroid of the parallel K-means clustering using the MSSA algorithm; and S4, obtaining and outputting the final clustering results. This method solves the problem of poor clustering results for high-dimensional data, achieving uniform data partitions, and improving the algorithm's sensitivity to initial centroids.

Description

Technical Field

[0001] The present invention relates to the field of big data mining, and in particular to a parallel K-means clustering method based on MapReduce and MSSA. Background Art

[0002] K-means clustering is an unsupervised learning method that divides data points into clusters based on certain similarity criteria, even in the absence of prior knowledge. It maximizes the similarity of data points within a cluster while minimizing the similarity between clusters, thereby discovering unknown groups within the data. It is widely used in machine learning, image analysis, pattern recognition, data compression, and other fields.

[0003] With the advent of the big data era, data volumes are growing rapidly. The data platforms of large internet companies receive petabytes of data daily, and trade transaction data and user rating data often have hundreds or even thousands of dimensions. This massive amount of high-dimensional data has exacerbated the performance issues of the K-means algorithm and severely restricted its application in big data environments. Therefore, improving the performance of the K-means algorithm in big data environments has become a major concern both domestically and internationally.

[0004] With the widespread adoption of the K-means algorithm within distributed frameworks, the MapReduce big data computing model developed by the Apache Foundation has gained widespread favor among researchers due to its high stability and strong scalability. Specifically, Sayer et al. proposed MR-GAPKCA, an algorithm that combines a genetic algorithm with K-means within the MapReduce big data framework. This algorithm optimizes K-means centroid selection by incorporating genetic algorithm operations such as population selection, crossover, and mutation, thereby improving K-means' global search capabilities. Furthermore, within the MapReduce framework, MR-GAPKCA implements parallel clustering by computing the distances between samples and cluster centers in parallel using the Map function, then merging identical clusters within a partition using the Combine function. Finally, MR-GAPKCA merges all samples within the same cluster using the Reduce function. While MR-GAPKCA outperforms the traditional K-means algorithm in both runtime and clustering performance, it still suffers from three drawbacks: poor clustering performance with high-dimensional data, uneven data partitioning, and sensitivity to initial centroids. Summary of the Invention

[0005] The present invention aims to at least solve the technical problems existing in the prior art, and particularly innovatively proposes a parallel K-means clustering method based on MapReduce and MSSA.

[0006] In order to achieve the above object of the present invention, the present invention provides a parallel K-means clustering method based on MapReduce and MSSA, comprising the following steps:

[0007] S1, feature weighting and feature extraction for high-dimensional data;

[0008] S2, divides the data after feature extraction, selects high-quality partition points through two-segment mapping, and divides the data into generalized hyperplanes according to the partition points to obtain uniform partitions on the map;

[0009] S3, using the MSSA algorithm to optimize the centroid of parallel K-means clustering;

[0010] S4, obtain the final clustering result and output it.

[0011] Furthermore, the S1 includes the following steps:

[0012] S1-1, feature weighting: First, obtain the feature set I and weight set w of the initial data set X, then calculate the Kendall correlation coefficient between the features, and finally obtain the weighted data set X according to the Kendall correlation coefficient weight KCCW. w ;

[0013] S1-2, feature extraction: Get the weight dataset X w After that, first construct DSAE and determine the loss function L DSAE , and then use the adaptive penalty coefficient λ to adjust L DSAE The penalty term, finally w Feature extraction is performed to obtain a low-dimensional dataset X′.

[0014] The DSAE construction includes: first, initializing the weights W of the coding layer and the bias b of the hidden layer, encoding and compressing the input sample X through the coding layer to obtain the hidden layer representation. Then, at the decoding layer, the Sigmoid function is selected as the decoding layer activation function to decode and reconstruct the hidden layer representation.

[0015] Furthermore, the Kendall correlation coefficient weight KCCW includes:

[0016]

[0017] Among them, KCCW(I i ) represents feature I i Kendall correlation coefficient weights;

[0018] Z(I i ) represents KCC(I i ,I j )’s average value;

[0019] Z(Ij ) is the difference between feature I i The mean of the Kendall correlation coefficient for any other feature;

[0020] KCC(I i ,I j ) is feature I i With Feature I j Kendall correlation coefficient;

[0021] d is the characteristic dimension of the data.

[0022] Furthermore, the loss function L DSAE include:

[0023]

[0024] Among them L DSAE (W, b) represents the loss function of the encoding layer weight W and the hidden layer bias b;

[0025] L(·) is the square error function;

[0026] is the reconstruction of the input sample X;

[0027] λ is the penalty coefficient;

[0028] k i,j For input x i Corresponding to the activation value of hidden layer node j, input x i is the i-th sample;

[0029] ||·||1 denotes the L1 norm.

[0030] Furthermore, the adaptive penalty coefficient λ penalty coefficient includes:

[0031]

[0032] Among them S i is the input feature I z The variance of the above data;

[0033] n is the number of data;

[0034] c is a constant.

[0035] Furthermore, the S2 includes the following steps:

[0036] S2-1, selection of partition points: First, select the support point set S m , and use the support point set S m Map the data set to the h-dimensional vector space, then perform SFC mapping on the vector space, and calculate the SFC(r v) value to select the partition point and get the partition point set S c ;

[0037] S2-2, data partitioning: the partition point set S obtained based on the two-segment mapping c , proposed the generalized hyperplane partition set P i r Divide the original dataset evenly.

[0038] Furthermore, the S2-1 includes the following steps:

[0039] 1) Sampling: First, randomly sample the data set R to obtain the sample set R s ;

[0040] 2) Support point set selection: According to the support point set quality evaluation function DPP(S m ) as the evaluation index, from the sample set R s Select DPP(S m ) The largest set, that is, the first three data points closest to 1, constitute the support point set S m ;

[0041] The support point set quality evaluation function DPP(S m ) is as follows:

[0042]

[0043] where r i ,r j ∈R s , is the sample set R s Any two points in ;

[0044] d(r i ,r j ) o is the distance between two points in the original space;

[0045] d(r i ,r j ) v is the distance between two points in the mapping space;

[0046] g represents the number of sampling samples;

[0047] 3) Support point mapping: Map the data point r(r∈R s ) through the support point set S m ={s m1 ,s m2 ,...,s mh} mapping, transforming to a new data point r in h-dimensional vector space v , r v From r to S mThe h-dimensional vector composed of distance is represented by: r v = <d(r,s m1 ),d(r,s m2 ),...,d(r,s mh )>;where d(r,s m1 ) represents r to s m1 The Euclidean distance, d(r,s m2 ) represents r to s m2 The Euclidean distance, d(r,s mh ) represents r to s mh The Euclidean distance of

[0048] 4) SFC mapping: Use SFC mapping to map r v Mapped to integer value SFC(r v ), and according to SFC(r v ) to obtain the partition point.

[0049] Furthermore, the generalized hyperplane partition set P i r include:

[0050]

[0051] Where r represents a data point;

[0052] R represents the dataset R;

[0053] Indicates existence;

[0054] ∧ represents the logical "and";

[0055] s ci is the i-th partition point;

[0056] s cj for any other partition point;

[0057] d(r,s ci ) represents r to the partition point s ci The Euclidean distance of

[0058] d(r,s cj ) represents r to the partition point s cj The Euclidean distance.

[0059] Furthermore, the MSSA algorithm includes:

[0060] S3-1, Leading Sparrow Search: Proposing Non-uniform Leading Sparrow X i ' j ,Using non-uniform mutation operators to perturb the positions of leading sparrows and increase the diversity of the population;

[0061] S3-2, follower sparrow following: using the adaptive coefficient to improve the follower sparrow's following step length, and propose an adaptive follower sparrow Updated the position of the follower Sparrow.

[0062] Furthermore, the non-uniform variation leads to Sparrow X i ' j include:

[0063]

[0064] where X ij represents the j-dimensional component of the i-th sparrow;

[0065] Δ(t,·) is the non-uniform mutation operator;

[0066] t is the number of iterations,

[0067] UB,LB are X ij The upper and lower bounds of are both constants;

[0068] ξ is a random number of 0 or 1;

[0069] The adaptive follower sparrow include:

[0070]

[0071] Among them, Q is a random number that obeys the normal distribution;

[0072] exp(·) is an exponential function with the natural constant e as its base;

[0073] Indicates the worst position at the tth iteration;

[0074] represents the data volume of the j-dimensional component of the i-th sparrow at the t-th iteration;

[0075] i means the current data is the i-th data;

[0076] n represents the population size, which is the amount of data;

[0077] represents the best position of the tth iteration;

[0078] α t is a variable that decreases with respect to the number of iterations t

[0079] γ is a vector of two random numbers in [0,1];

[0080] C is the coefficient vector;

[0081] represents the best position of the tth iteration;

[0082] |·| indicates an absolute value.

[0083] In summary, due to the adoption of the above technical solution, the present invention has the following advantages:

[0084] (1) A dimensionality reduction strategy DRKCAE based on Kendall correlation coefficient and DSAE is designed. First, the Kendall correlation coefficient weight KCCW is proposed to weight the high-dimensional data features to eliminate the interference of irrelevant features. Then, a DSAE based on L1 norm and adaptive penalty coefficient is constructed for feature extraction to obtain compact low-dimensional data, which solves the problem of poor clustering effect when facing high-dimensional data.

[0085] (2) Design a generalized hyperplane partitioning strategy UPS based on two-segment mapping to partition the data. First, use the two-segment mapping to obtain the partition point set S c , then proposed the generalized hyperplane partition set P i r , and according to S c Use P i r Divide the data to get uniform data partitions.

[0086] (3) A non-uniform mutation sparrow search algorithm (MSSA) was proposed to optimize the centroid selection of the parallel K-means algorithm, which improved the problem that the algorithm was sensitive to the initial centroid.

[0087] Additional aspects and advantages of the present invention will be set forth in part in the description which follows and, in part, will be obvious from the description which follows, or may be learned by practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0089] Figure 1 It is a schematic flow diagram of the present invention.

[0090] Figure 2 This is a schematic diagram of the present invention comparing the NMI values ​​of four clustering methods under four data sets.

[0091] Figure 3 It is a schematic diagram of the changes in the parallel efficiency experiments of four clustering methods under four data sets in the present invention. DETAILED DESCRIPTION

[0092] The following describes embodiments of the present invention in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only to explain the present invention and are not to be construed as limiting the present invention.

[0093] The method can be applied in the following fields: machine learning, image analysis, pattern recognition, data compression, data analysis and so on.

[0094] When applied in the image field, the specific implementation steps are as follows:

[0095] S1, feature weighting and feature extraction for high-dimensional image data;

[0096] S2, divides the image data after feature extraction, selects high-quality division points through two-segment mapping, and divides the data into generalized hyperplanes according to the division points to obtain uniform partitions on the map;

[0097] S3, using the MSSA algorithm to optimize the centroid of parallel K-means clustering;

[0098] S4, obtain the final clustering result and output it.

[0099] Image analysis research generally includes content-based image retrieval, face recognition, expression recognition, optical character recognition, handwriting recognition, medical image analysis, and video object extraction. When applied to image retrieval, the clustering results are the content corresponding to the image; when applied to medical image analysis, the clustering results are disease diagnosis results.

[0100] The present invention provides a parallel K-means clustering method based on MapReduce and MSSA, such as Figure 1 As shown, the following steps are included:

[0101] S1, design DRKCAE strategy to perform feature weighting and feature extraction on high-dimensional data to obtain low-dimensional high-quality data;

[0102] S2 proposes a UPS strategy to partition the data after feature extraction. High-quality partition points are selected through two-segment mapping, and the data is partitioned by generalized hyperplanes based on the partition points to obtain uniform partitions on the map.

[0103] S3, combining non-uniform mutation operator and adaptive step size, proposed the MSSA algorithm, and used the MSSA algorithm to optimize the centroid of parallel K-means clustering;

[0104] S4, obtain the final clustering result and output it.

[0105] The present invention effectively improves the clustering effect and parallel efficiency of the parallel k-means algorithm. The details of the present invention are as follows.

[0106] 1. Data Dimensionality Reduction

[0107] At present, when the K-means algorithm based on MapReduce is used to cluster high-dimensional data, due to the existence of irrelevant features and abnormal sparse structure in high-dimensional data, the interference of irrelevant features and the failure of measurement methods together lead to poor clustering effect. In order to solve this problem, MR-MSKCA designed the DRKCAE strategy to process high-dimensional data, which not only reduces the interference of irrelevant features on clustering, but also obtains a compact data representation through nonlinear dimensionality reduction. The specific process of this strategy is divided into two steps: (1) Feature weighting: First, obtain the feature set I and weight set w of the initial data set X, then calculate the Kendall correlation coefficient between the features, and finally propose the Kendall correlation coefficient weight KCCW, and obtain the weighted data set X according to KCCW. w (2) Feature extraction: After obtaining the weighted dataset X w After that, first construct DSAE and determine the loss function L DSAE , then design the adaptive penalty coefficient λ to adjust L DSAE The penalty term, finally w Feature extraction is performed to obtain a low-dimensional dataset X′.

[0108] 1.1 Feature Weighting

[0109] Before feature extraction, the features of the initial data set are weighted to reduce the influence of irrelevant features on the clustering results. The steps of feature weighting are as follows: First, obtain the initial data set X, and record the features and feature weights of the initial data set as I = [I1, I2, ..., I d ] and w=[w1,w2,...,w d ], and [w1,w2,...,w d ] are all initialized to 1. Then calculate I1 and [I2,I3...,I d ], and after calculating I1, the other features are calculated in turn. Finally, the Kendall correlation coefficient weight KCCW is proposed to calculate [I1,I2,...,I d ] obtains the new feature weight w z =[w z1 ,w z2 ,...,w zd ], and according to w z Calculate the new feature I z =[w z1 I1,w z2 I2,...,w zd Id ], get the weighted dataset X w .

[0110] Theorem 1 (Kendall correlation coefficient weight KCCW) Let KCC(I i ,I j ) is feature I i The Kendall correlation coefficient with any other feature, d is the feature dimension of the data, then feature I i The value of KCCW is:

[0111]

[0112] Where Z(I i ) represents KCC(I i ,I j )’s average value;

[0113] Z(I j ) is the difference between feature I i The mean Kendall correlation coefficient for any other feature.

[0114] Proof: Known Kendall correlation coefficient KCC (I i ,I j ) size represents I i The Kendall correlation coefficient with any other feature, KCC(I i ,I j ) is larger, the greater the similarity between features is. Therefore, KCC(I i ,I j ) the average value Z(I i ) represents I i The degree of correlation with all features. i ) into KCCW(I i ) is calculated by the formula, Z(I i ) and ΣZ(I j ) reflects the ratio of Z(I i ) in the total correlation, the larger the value, the more I i The more important, KCCW(I i ) is a good measure of feature I i The importance of can be used as the weight of the Kendall correlation coefficient for feature weighting. The proof is complete.

[0115] 1.2 Feature Extraction

[0116] After obtaining weighted features, the influence of irrelevant features is suppressed. DSAE is then used for feature extraction, resulting in a low-dimensional, compact representation of the original data. This mitigates the impact of high-dimensional data structure sparsity on clustering. The feature extraction process consists of three main steps: DSAE construction, penalty coefficient determination, and feature extraction.

[0117] (1) DSAE structure

[0118] The structure of the deep sparse autoencoder DSAE consists of an encoding layer, a decoding layer, and multiple hidden layers. First, the weights W of the encoding layer and the bias b of the hidden layer are initialized. The input sample X is encoded and compressed through the encoding layer to obtain the hidden layer representation. Then, the Sigmoid function is selected as the decoding layer activation function in the decoding layer to decode and reconstruct the hidden layer representation. Finally, the loss function is determined. In order to extract the sparse feature structure, this paper proposes a loss function L based on the L1 norm. DSAE Effectively suppress the hidden layer nodes.

[0119] Theorem 2 (Loss function L based on L1 norm DSAE ): Known is the reconstruction of the input sample X, L(·) is the square error function, k i,j For input x i Corresponding to the activation value of hidden layer node j, input x i is the i-th sample, x i ∈X; λ is the penalty coefficient, then the loss function L based on the L1 norm DSAE for:

[0120]

[0121] Among them L DSAE (W, b) represents the loss function of the encoding layer weight W and the hidden layer bias b;

[0122] ||·||1 denotes the L1 norm.

[0123] (2) Determination of penalty coefficient

[0124] After constructing the loss function L DSAE Finally, considering that the features of sample input vary greatly, the L1 penalty term with a fixed λ cannot adapt to all features. Therefore, this paper proposes an adaptive penalty coefficient λ to adjust the penalty term. According to the distribution of different features, the penalty coefficient is adaptively adjusted to achieve the best penalty effect.

[0125] Theorem 3 (Adaptive penalty coefficient λ): Given the input feature I z The variance of the above data is S i , the number of data is n, c is a constant, then the adaptive penalty coefficient λ is:

[0126]

[0127] Proof: Since S i It can reflect the distribution of data under this feature. The more information S i The larger the value, the less information S i The smaller the adaptive penalty coefficient λ is, the more information-rich the feature is. The larger the adaptive penalty coefficient λ is, the greater the penalty on the hidden layer nodes, and irrelevant features are quickly suppressed. Therefore, the adaptive penalty coefficient λ can be used as the penalty coefficient for the activation of hidden layer nodes. The proof is complete.

[0128] (3) Feature extraction

[0129] Taking the first layer of sparse autoencoder as an example, firstly weight the feature weighted dataset X n*d Through the forward pass training of the encoding layer of DSAE, the reconstructed samples are obtained. Then the loss function L is minimized. DSAE Perform backpropagation training for the training target, and continuously optimize the weight W and hidden layer bias b. Finally, the hidden layer representation calculated by W and b is used as the input X of the next layer. (1) .X (1) Through the remaining t-1 layers of sparse autoencoders, training layer by layer, the encoding of the t-th hidden layer is obtained as the final result X′, which is the dataset after data extraction.

[0130] 2. Data Partitioning

[0131] At present, the K-means algorithm based on MapReduce is prone to data skew due to the differences in input data distribution. Direct data partitioning can affect clustering efficiency. In order to solve the problem of uneven data partitioning in the data partitioning stage, this paper proposes a UPS strategy, which designs two-segment mapping to obtain high-quality partitioning points and uses them to evenly partition the input data set. The strategy is divided into two steps: (1) Partitioning point selection: First, the support point set precision evaluation function DPP (S m ) Select the support point set S m , and use the support point set S m Map the data set to the h-dimensional vector space, then perform SFC mapping on the vector space, and calculate the SFC(r v ) value to select the dividing point S c (2) Data partitioning: The high-quality partition points S obtained based on the two-segment mapping c , proposed the generalized hyperplane partition set P i r Divide the original dataset evenly.

[0132] 2.1 Selection of division points

[0133] To achieve a uniform partitioning of a dataset, a two-segment mapping is first performed on the dataset to obtain high-quality partitioning points. These partitioning points are then used to evenly partition the data. This two-segment mapping, which includes support point mapping and space-filling curve (SFC) mapping, maps the data in the original metric space into a one-dimensionally encoded SFC space, ensuring similarity between the spaces before and after the mapping. Therefore, partitions between one-dimensional integers in the SFC space can be used to estimate partitions between the original data. The steps for selecting partitioning points in the two-segment mapping process include sampling, support point set selection, support point mapping, and SFC mapping.

[0134] (1) Sampling

[0135] In order to reduce the support point mapping cost, we first randomly sample the data set R and obtain the sample set R s The purpose of sampling is to reduce the time cost of supporting point mapping and facilitate the further selection of partitioning points without affecting the partitioning of the entire data set R.

[0136] (2) Support point set selection

[0137] After sampling, the support points required for mapping are selected. In order to keep the vector space as similar as possible to the original space, it is necessary to select the support points from the sampling set R. s Get a high-quality support point set S m Therefore, the support point set quality evaluation function DPP(S m ) as the evaluation index, and select DPP(S m )The largest set is S m .

[0138] Theorem 4 (Support point set precision function DPP(S m )):Assume S m is the support point set, r i ,r j ∈R s ,d(r i ,r j ) o is the distance between two points in the original space, d(r i ,r j ) v is the distance between two points in the mapping space, then DPP(S m ) function is:

[0139]

[0140] Where g represents the number of sampling samples.

[0141] Proof: According to the triangle inequality relationship d(r i ,r j ) o ≥max{|d(ri ,s mt )-d(r j ,s mt )||s mt ∈S m}=d(r i ,r j ) v It can be seen that d(r i ,r j ) o Provided d(r i ,r j ) v The upper bound of d(r i ,r j ) v The closer to d(r i ,r j ) o , the higher the similarity between the vector space and the original space, the better the support point mapping effect, that is, S m Conversely, the lower the similarity between the vector space and the original space, the worse the support point mapping effect, that is, S m Therefore, S m The accuracy can be obtained by d(r i ,r j ) v and d(r i ,r j ) o The average ratio is evaluated and the proof is completed.

[0142] (3) Support point mapping

[0143] According to DPP(S m ) Get the support point set S m , then the sample data set R s Perform support point mapping. Data point r(r∈R s ) through the support point set S m ={s m1 ,s m2 ,...,s mh} mapping, transforming to a new data point r in h-dimensional vector space v , r v From r to S m The h-dimensional vector composed of distance is represented by: r v = <d(r,s m1 ),d(r,s m2 ),...,d(r,s mh )>.

[0144] (4) SFC Mapping

[0145] After support point mapping, use SFC mapping to map r v Mapped to integer value SFC(r v ), and according to SFC(r v ) to obtain the division point. The specific process is: first construct the SFC curve of h-dimensional space for r v Execute the mapping, each r v Get the corresponding integer code value SFC(r v Then, the code values ​​are grouped according to the number of data partitions a, and the total SFC (r v ) is divided into a parts. Finally, the first SFC (r v ) The corresponding data points enter the partition point set S c .

[0146] 2.2 Data Partitioning

[0147] The partition point set S is obtained c Afterwards, due to S c It is selected by evenly partitioning the integer domain after the two mappings, and the two mappings maintain the similarity of the space, so S c Divide the original space into uniform partitions. The specific process is as follows: For each data point r (r∈R), calculate r to each partition point s c (s c ∈S c ) of the Euclidean distance. Then the generalized hyperplane partition set P is proposed i r , cluster the adjacent data points r to form P i r Data set. When all data points r are calculated, that is, r enters the corresponding partition P i , the division is completed, and a data partition P is obtained i .

[0148] Theorem 5 (Generalized hyperplane partition set P i r ) Let r(r∈R) be a point in the data set, s ci is the i-th partition point, s cj For any other partition point, d(r,s ci ),d(r,s cj ) corresponds to the Euclidean distance from r to different partition points, then P i r for:

[0149]

[0150] Where ∧ represents the logical "and".

[0151] Proof: By the condition d(r,s ci )≤d(r,s cj ) It can be seen that the data set R to the partition point s ci The group of points r with the closest Euclidean distance constitutes P i r . The Euclidean distance is a similarity measure, that is, P i r and partition point S ci The greater the similarity, P i r The set constitutes the partition P i All points of . So P i r The selection process is to divide the space where the data set R is located into hyperplanes, which can be expressed by P i r Divide the data evenly and the proof is complete.

[0152] 3. Parallel Clustering

[0153] After evenly partitioning the data, the partitioned data blocks are fed into a MapReduce task for parallel clustering. The current MapReduce-based K-means algorithm requires initializing the centroids when performing parallel clustering. However, improper initial centroids can cause the algorithm to fall into a local optimum, making the algorithm very sensitive to the selection of the initial centroids. Because the SSA algorithm has excellent global search and optimization capabilities, the SSA algorithm was considered to improve the parallel K-means algorithm's sensitivity to initial centroids. However, the algorithm's diversity gradually decreases during iterations, potentially leading to premature convergence. To address this issue, this paper proposes a non-uniform mutation sparrow search algorithm (MSSA). This algorithm then uses MSSA to optimize the centroid optimization process and obtain the final clustering results.

[0154] 3.1 MSSA Algorithm

[0155] The main implementation steps of the MSSA algorithm are: (1) Leading sparrow search: Propose a non-uniform leading sparrow X′ ij , using non-uniform operators to perturb the position of the leading sparrow and increase the diversity of the population. (2) Follower sparrow following: using the adaptive coefficient to improve the following step length of the follower sparrow, an adaptive follower sparrow is proposed , update the position of the follower sparrow.

[0156] (1) Leading Sparrow Search

[0157] In the SSA algorithm, the leading sparrow leads the entire population to move. When the warning value is reached, it will lead the sparrow population to search again, which can effectively jump out of the local optimum. However, before the warning value is reached, the diversity will gradually decrease with the iteration of the algorithm. In order to make the algorithm always maintain the global search capability, a non-uniform mutation leading sparrow X is proposed.i ' j , leading Sparrow X to generate ij The j-th dimension component of is subjected to non-uniform mutation.

[0158] Theorem 6 (Non-uniform mutation leads to sparrow X′ ij ): It is known that Δ(t,y) is a non-uniform mutation operator, UB and LB are respectively ij The upper and lower bounds of t are the number of iterations, and non-uniform mutation leads to sparrow X′ ij for:

[0159]

[0160] Where UB, LB are constants, X ij represents the j-dimensional component of the i-th sparrow;

[0161] ξ is a random number of 0 or 1.

[0162] Proof: By It is easy to see that as the number of iterations t increases, Getting bigger and bigger, As the value of Δ(t,y) decreases, the return value of Δ(t,y) gradually decreases from y to 0. In other words, at the beginning of the iteration, X′ ij is severely disturbed and has strong global search capabilities. As the algorithm continues to evolve, X′ ij The search radius is reduced, which avoids adverse mutations and speeds up the convergence speed, so non-uniform mutation leads to sparrow X′ ij Valid, proven.

[0163] (2) Follow the sparrow

[0164] In the SSA algorithm, the follower sparrow jumps directly to the vicinity of the current optimal solution. When the leading sparrow is in a bad position, the algorithm quickly converges to the local optimum, which is not conducive to the algorithm obtaining the global optimum. Therefore, the adaptive coefficient is used to improve the process of the follower sparrow following, and the adaptive follower sparrow is proposed.

[0165] Theorem 7 (Adaptive Follower Sparrow ) The best position of the tth iteration is known A=2α t γ-α t is the adaptive coefficient, α t It is a variable that decreases with the number of iterations t, linearly decreasing from 2 to 0, γ is a vector of two random numbers in [0,1], C is a coefficient vector, then the adaptive follower sparrow for:

[0166]

[0167] Among them, Q is a random number that obeys the normal distribution;

[0168] X worst It is the worst position at present;

[0169] n represents the population size, which is the amount of data;

[0170] i means the current data is the i-th data;

[0171] represents the data volume of the j-dimensional component of the i-th sparrow at the t-th iteration.

[0172] Proof: Because α t As the number of iterations t increases, it decreases, so the adaptive coefficient A = 2α t γ-α t It also decreases with the increase of the number of iterations t. At the beginning of the iteration, A is large. Also larger, following the position of the sparrow Towards The approach is not active and has a strong randomness, which meets the need for global search capabilities in the early stages of the algorithm. As the algorithm continues to evolve, the position of the sparrow Gradually follow up It avoids adverse mutations while speeding up the convergence speed. Valid, proven.

[0173] The MSSA algorithm process is as follows:

[0174] ① Initialization of sparrow population.

[0175] ②Calculate the fitness of the sparrow population and select the best and worst positions.

[0176] ③ Select the leading sparrow and the following sparrow based on their fitness. Update the leading sparrow's position and perturb it using Theorem 6. Then, use Theorem 7 to update the following sparrows, and finally update the position of the warning sparrow.

[0177] ④Recalculate the sparrow's fitness.

[0178] ⑤ Determine whether the end condition is met. If not, turn to ③. If it is met, the algorithm ends.

[0179] 3.2 Parallel Clustering

[0180] After the MSSA is proposed, the good global search and optimization capabilities of MSSA can be combined with parallel K-means to process data and ultimately achieve parallel clustering. The specific steps are as follows:

[0181] (1) Consider the data set as a group of sparrows S = {s1,s2,...,sn Initialize the basic parameters: population size N, number of leading sparrows PD, number of warning sparrows SD, warning value R2, and maximum number of iterations T.

[0182] (2) Parallel partitioning and clustering are performed in the MapReduce framework. In the Map phase, the index of the leading sparrow is used as the key, and the Euclidean distance from other sparrows to the leading sparrow is calculated in parallel as the value, and the components are<Key,Value> The key-value pairs are passed to the Reduce task. The Reduce stage calculates the center of mass of each sparrow based on the Map result.

[0183] (3) The fitness of each sparrow is calculated in parallel using the MapReduce framework. The fitness is calculated as the inverse of the distance from the sparrow to the center of mass. When calculating the fitness, the index of the leading sparrow is used as the key and the fitness is used as the value. The sparrows are sorted according to their fitness.

[0184] (4) Select the first PD sparrow with the best fitness as the leading sparrow, and the rest as the following sparrows. After updating the position of the leading sparrow in parallel, use non-uniform mutation to lead the sparrow X′ ij Perturbation, and then according to the adaptive follower sparrow Update the position of the follower sparrow in parallel.

[0185] (5) Randomly select SD sparrows as warning sparrows, use the warning value R2 as the judgment condition to warn the population, and update the position in parallel.

[0186] (6) After one iteration is completed, determine whether the maximum number of iterations has been reached. If not, go to step (2). When the maximum number of iterations is reached, the algorithm ends and the final clustering result is output.

[0187] 4. Validity of MR-MSKCA

[0188] To validate the performance of MR-MSKCA, we compared four clustering algorithms: MR-MSKCA, MR-KNMF, MR-PGDLSH, and MR-GAPKCA, on the Drug Review, Covertype, Daily Sport, and DailySport+ datasets. The experimental datasets are shown in the table below.

[0189]

[0190] 4.1 Clustering Effect of MR-MSKCA Method

[0191] In order to verify the clustering effect of MR-MSKCA, this paper compares the NMI values ​​of MR-MSKCA, MR-KNMF, MR-PGDLSH, and MR-GAPKCA in four datasets: Drug Review, Covertype, Daily Sport, and Daily Sport+. The experimental results are as follows: Figure 2 shown.

[0192] Experiments show that while the NMI value decreases with increasing dataset feature dimensionality, the MR-MSKCA algorithm consistently performs best, and this trend increases with increasing feature dimensionality and dataset size. On the Drug Review dataset, which has a small feature dimensionality, the MR-MSKCA algorithm's NMI value is 2.0%, 2.8%, and 3.2% higher than MR-KNMF, MR-PGDLSH, and MR-GAPKCA, respectively, showing a relatively insignificant difference. On the Covertype dataset, which has a moderate feature dimensionality, the MR-MSKCA algorithm's NMI value is 6.2%, 7.5%, and 9.6% higher than MR-KNMF, MR-PGDLSH, and MR-GAPKCA, respectively, showing a gradual gap. On the Daily Sport+ dataset, which has a large feature dimensionality, the MR-MSKCA algorithm's NMI value is 19.2%, 22.8%, and 24% higher than MR-KNMF, MR-PGDLSH, and MR-GAPKCA, respectively, showing a significant difference. The reasons for this phenomenon are: (1) MR-MSKCA designed the DRKCAE strategy to reduce the dimensionality of the data, effectively reducing the dimensionality of high-dimensional data, avoiding the influence of irrelevant features, and improving the clustering effect; (2) The MR-MSKCA algorithm uses MSSA to avoid centroid sensitivity, enhance the global search ability of the algorithm, and further improve the clustering effect. On datasets with small feature dimensions, the dimension and initial centroid have little effect on the clustering effect, resulting in limited improvement in clustering effect and little difference in NMI values. On datasets with large feature dimensions, Daily Sport and Daily Sport+, the NMI value of MR-MSKCA is significantly higher than that of the other three algorithms. At the same time, it shows that MR-MSKCA has better clustering effect through the DRKCAE strategy and MSSA optimization, and exhibits better clustering performance than the other three algorithms.

[0193] 4.2 Parallel efficiency of the MR-MSKCA method

[0194] To verify the parallelization performance of MR-MSKCA, this paper experiments on MR-MSKCA, MR-KNMF, MR-PGDLSH, and MR-GAPKCA on four datasets: Drug Review, Covertype, Daily Sport, and Daily Sport+. The experimental nodes are added one by one, and the changes in the algorithm acceleration ratio with different numbers of nodes are compared.

[0195] Experimental data Figure 3 As shown:

[0196] Depend on Figure 3 It can be seen that the speedup ratios of MR-MSKCA, MR-KNMF, MR-PGDLSH, and MR-GAPKCA algorithms all increase with the number of nodes in the four datasets, among which MR-MSKCA has the most obvious increase trend. Moreover, compared with other algorithms, MR-MSKCA maintains the largest speedup ratio on any dataset. When four nodes are operating simultaneously, the speedup ratios of MR-MSKCA algorithm on Drug Review dataset are 0.18, 0.13, and 0.27 higher than those of the other three algorithms; on Covertype dataset, the speedup ratios of MR-MSKCA algorithm are 0.57, 0.43, and 0.68 higher than those of the other three algorithms; on Daily Sport dataset, the speedup ratios of MR-MSKCA algorithm are 0.24, 0.09, and 0.27 higher than those of the other three algorithms; and on Daily Sport+ dataset, the speedup ratios of MR-MSKCA algorithm are 0.33, 0.24, and 0.37 higher than those of the other three algorithms. The reasons for these phenomena are: (1) MR-MSKCA algorithm uses UPS strategy to evenly divide the dataset, ensuring load balancing of nodes. (2) The MR-MSKCA algorithm uses the DRKCAE strategy to reduce dimensionality, avoiding the calculation of redundant features, reducing the amount of data interacting between nodes to a certain extent, and improving the parallel performance of the algorithm. As the number of nodes increases, the parallel advantage gradually increases, making the acceleration ratio growth trend of the MR-MSKCA algorithm more obvious than that of MR-KNMF, MR-PGDLSH, and MR-GAPKCA. However, MR-KNMF and MR-GAPKCA do not consider node load balancing, resulting in an imbalance in the amount of calculation between nodes, which leads to a decrease in the parallel efficiency of the algorithm. For MR-PGDLSH, due to its load balancing strategy, the parallel efficiency of MR-KNMF and MR-GAPKCA is improved compared with MR-KNMF. However, due to the limitations of the iterative method of its clustering process, the parallel performance of the algorithm is limited to a certain extent. In summary, the acceleration ratio performance of MR-MSKCA is better than that of the other three algorithms, and it shows good parallel performance on large-scale high-dimensional data sets.

[0197] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

Claims

1. A parallel K-means clustering method based on MapReduce and MSSA, characterized in that: The following steps are involved: S1, performing feature weighting and feature extraction on high-dimensional image data; S1 includes the following steps: S1-1, feature weighting: First, obtain the feature set I and weight set w of the initial data set X, then calculate the Kendall correlation coefficient between the features, and finally obtain the weighted data set X according to the Kendall correlation coefficient weight KCCW. w ; The Kendall correlation coefficient weight KCCW includes: Among them, KCCW(I i ) represents feature I i Kendall correlation coefficient weights; Z(I i ) represents KCC(I i ,I j )’s average value; Z(I j ) is the difference between feature I i The mean of the Kendall correlation coefficient for any other feature; KCC(I i ,I j ) is feature I i With Feature I j Kendall correlation coefficient; d is the characteristic dimension of the data; S1-2, feature extraction: Get the weight dataset X w After that, first construct DSAE and determine the loss function L DSAE , and then use the adaptive penalty coefficient λ to adjust L DSAE The penalty term, finally w Feature extraction is performed to obtain a low-dimensional dataset X′; the loss function L DSAE include: Among them L DSAE (W, b) represents the loss function of the encoding layer weight W and the hidden layer bias b; L(·) is the square error function; is the reconstruction of the input sample X; λ is the penalty coefficient; k i,j For input x i Corresponding to the activation value of hidden layer node j, input x i is the i-th sample; ‖·‖1 represents the L1 norm; The adaptive penalty coefficient λ penalty coefficient includes: Among them S i is the input feature I z The variance of the above data; n is the number of data; c is a constant; S2, divides the image data after feature extraction, selects high-quality division points through two-segment mapping, and divides the data into generalized hyperplanes according to the division points to obtain uniform partitions on the map; S3, using the MSSA algorithm to optimize the centroid of parallel K-means clustering; S4, obtain the final clustering result and output it.

2. The parallel K-means clustering method based on MapReduce and MSSA according to claim 1, characterized in that: The S2 comprises the following steps: S2-1, selection of partition points: First, select the support point set S m , and use the support point set S m Map the data set to the h-dimensional vector space, then perform SFC mapping on the vector space, and calculate the SFC(r v ) value to select the partition point and get the partition point set S c ; S2-2, data partitioning: the partition point set S obtained based on the two-segment mapping c , proposed the generalized hyperplane partition set P i r Divide the original dataset evenly.

3. The parallel K-means clustering method based on MapReduce and MSSA according to claim 2, characterized in that: The S2-1 includes the following steps: 1) Sampling: First, randomly sample the data set R to obtain the sample set R s ; 2) Support point set selection: According to the support point set quality evaluation function DPP(S m ) as the evaluation index, from the sample set R s Select DPP(S m ) The first three data points closest to 1 form the support point set S m ; The support point set quality evaluation function DPP(S m ) is as follows: where r i ,r j ∈R s , is the sample set R s Any two points in ; d(r i ,r j ) o is the distance between two points in the original space; d(r i ,r j ) v is the distance between two points in the mapping space; g represents the number of sampling samples; 3) Support point mapping: Map the data point r(r∈R s ) through the support point set S m ={s m1 ,s m2 ,...,s mh } mapping, transforming to a new data point r in h-dimensional vector space v ; 4) SFC mapping: Use SFC mapping to map r v Mapped to integer value SFC(r v ), and according to SFC(r v ) to obtain the partition point.

4. According to a parallel K-means clustering method based on MapReduce and MSSA in claim 2, the generalized hyperplane partition set P i r include: Where r represents a data point; R represents the dataset R; Indicates existence; ∧ represents the logical "and"; s ci is the i-th partition point; s cj for any other partition point; d(r,s ci ) represents r to the partition point s ci The Euclidean distance of d(r,s cj ) represents r to the partition point s cj The Euclidean distance.

5. The parallel K-means clustering method based on MapReduce and MSSA according to claim 1, wherein the MSSA algorithm comprises: S3-1, Leading Sparrow Search: Propose Non-uniform Leading Sparrow X′ ij ,Using non-uniform mutation operators to perturb the positions of leading sparrows and increase the diversity of the population; S3-2, follower sparrow following: using the adaptive coefficient to improve the follower sparrow's following step length, and propose an adaptive follower sparrow Updated the position of the follower Sparrow.

6. According to the parallel K-means clustering method based on MapReduce and MSSA in claim 5, the non-uniform variation leads to sparrow X′ ij include: where X ij represents the j-dimensional component of the i-th sparrow; Δ(t,·) is the non-uniform mutation operator; t is the number of iterations, UB,LB are X ij The upper and lower bounds of are both constants; ξ is a random number of 0 or 1; The adaptive follower sparrow include: Among them, Q is a random number that obeys the normal distribution; exp(·) is an exponential function with the natural constant e as its base; Indicates the worst position at the tth iteration; represents the data volume of the j-dimensional component of the i-th sparrow at the t-th iteration; i means the current data is the i-th data; n represents the population size, which is the amount of data; represents the best position of the tth iteration; α t is a variable that decreases with respect to the number of iterations t Υ is a vector of two random numbers in [0,1]; C is the coefficient vector; represents the best position of the tth iteration; |·| indicates an absolute value.

Images