A data processing method and device based on distributed computing model

Abstract

The present invention relates to a data processing method and device based on a distributed computing model, belonging to the field of data processing technology. The method comprises the following steps: using a hash algorithm to preliminarily partition data, identifying and adjusting abnormal partitions, and refining abnormal partitions through fine-grained partitioning; identifying nodes with the same time window and generating an access policy set, selecting the optimal access set for task allocation; selecting core objects, expanding cluster clusters through a density-based clustering algorithm, and merging the clustering results of all Map tasks. The present invention improves the uniformity of the amount of data processed by each Reducer through a two-stage partitioning strategy, and then uses a cluster analysis method to find the optimal access set of the current distributed processing server node for task allocation, thereby improving data processing efficiency and resource utilization. Finally, a density-based clustering algorithm is used to remove noise points, thereby improving the accuracy of the clustering results.

Description

Technical Field

[0001] The present invention belongs to the field of data processing technology, and in particular relates to a data processing method and device based on a distributed computing model. Background Art

[0002] With the rapid growth of data in the big data era, traditional single-machine computing models can no longer meet processing needs. Therefore, data processing methods based on distributed computing models have emerged. Distributed computing can effectively improve computing efficiency, especially when complex calculations such as cluster analysis are required on large amounts of data.

[0003] However, data processing methods based on distributed computing still face some challenges in practical applications: in a distributed environment, improper data division may cause some nodes to be overloaded, affecting the overall computing efficiency; in the clustering process, uneven data distribution may cause some nodes to have too heavy processing tasks; at the same time, noise points and outliers will affect the clustering results. Summary of the Invention

[0004] In order to solve the above-mentioned problems existing in the prior art, the present invention provides a data processing method and device based on a distributed computing model. Through a two-stage partitioning strategy, the uniformity of the amount of data processed by each Reducer is improved. Then, by adopting a clustering analysis method, the optimal access set of the current distributed processing server nodes is found for task allocation, thereby improving data processing efficiency and resource utilization. Finally, noise points are removed through a density-based clustering algorithm, thereby improving the accuracy of the clustering results.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A first aspect of the present disclosure provides a data processing method based on a distributed computing model, comprising the following steps:

[0007] S1. Data collection and preprocessing: Collect various forms of data from the Internet of Things and preprocess the collected data, including data cleaning and conversion, removing noise data, and reducing data dimensionality;

[0008] S2. Data Partitioning and Partitioning: Use a hash algorithm to perform preliminary data partitioning, calculate the load of each Reducer, identify and adjust abnormal partitions, refine abnormal partitions through fine-grained partitioning, and dynamically adjust the partitioning strategy.

[0009] S3, dynamic task allocation: Based on the time window of the server node, identify the nodes with the same time window and generate an access policy set, and select the optimal access set for task allocation;

[0010] S4, Map and Reduce phase: Select core objects, expand clusters using a density-based clustering algorithm, merge the clustering results of all Map tasks, and use an iterative noise point processing strategy;

[0011] S5. Clustering result evaluation: Use evaluation indicators to evaluate clustering accuracy, analyze job execution time and load deviation, and verify the effectiveness of partitioning strategy and dynamic task allocation;

[0012] S6. Visualization: Select appropriate visualization tools to display the clustering results, design chart types based on the clustering results, and display the clustering results in a graphical way.

[0013] Furthermore, the data collection and preprocessing includes the following steps:

[0014] S11. Data cleaning and conversion: Identify missing values in the data and fill them by taking the mean; identify and delete duplicate records; use statistical methods to identify and handle outliers; standardize numerical data; and convert categorical variables to numerical data.

[0015] S12. Clean up the noisy data: Select the best K value through cross-validation. For each data point in the dataset, calculate its distance to other data points and identify its K nearest neighbors.

[0016] Check the distribution of each category label in the K neighbors or calculate the average value of the K neighbors. If the similarity between a data point and its neighbors is significantly lower than that of most data points, or the frequency of the category label of the point in the neighbors is significantly lower, then the data point is judged as noise and removed.

[0017] S13. Reduce the dimensionality of the data: Represent the original data set as a matrix, perform singular value decomposition on the data matrix, select the first K principal components based on the size of the singular values, and reconstruct the data set using the selected first K principal components.

[0018] Furthermore, the data division and partitioning includes the following steps:

[0019] S21, first stage hash partitioning:

[0020] Use the traditional hash algorithm to process the input data and map the data to R partitions based on the key value. Each partition corresponds to a Reducer, and the data is distributed to different partitions.

[0021] Set up a data counter on each Mapper node to track the size of the data in each partition and calculate the data load received by each Reducer L r :

[0022] ;

[0023] Where, x ij Indicates the i Mapper j Whether the partition is pulled, p ij Indicates the size of the partition; M Indicates the number of Mappers, and P indicates the number of partition types contained in each Mapper;

[0024] Set a threshold value. r The amount of data for each Reduce Lr If the value exceeds 90% of the threshold, it is considered an abnormal partition. Otherwise, it will be pulled normally by the Reducer for calculation and processing.

[0025] Furthermore, the data division and partitioning further includes the steps of:

[0026] S22, second stage hash partitioning:

[0027] Based on the abnormal partitions identified in the first phase, each abnormal partition is subdivided into b groups; the current load capacity of each Reducer is calculated, and the Reducer with the best current load capacity is selected for each subdivided group using a greedy algorithm;

[0028] Calculate the offset between the current optimal Reducer and the original partition number of the group , send the offset value to the Partitioner function, dynamically modify the hash partition function, and the modified Partioner function is:

[0029] ;

[0030] Where, is the modified Partitioner function, is the original Partitioner function;

[0031] S23. Merge partition results: Merge the partition results of the first stage with the partition results of the second stage to form a final partition file. The partition file contains the final partition information corresponding to each key value. According to the final partition file, the overall data is directly assigned to each Reducer for processing.

[0032] Furthermore, the dynamic task allocation includes the following steps:

[0033] Establishing an original set: Establishing an original set that includes all distributed processing server nodes;

[0034] Baseline node selection: The first server node in the set is used as the base, compared with other nodes, and all other server nodes that have a common time window with the base node are selected to form a new set;

[0035] Intersection calculation: among the selected server nodes, continue to calculate the intersection of the time windows, gradually expanding the set until all server nodes related to the benchmark node are found;

[0036] Set update: After completing the processing of a benchmark node, remove the node from the original set, continue to select the next node as the new benchmark, and repeat the above process until all nodes are processed to generate the final multiple access policy sets.

[0037] Furthermore, the dynamic task allocation further includes the steps of:

[0038] Select an access policy set: From the generated access policy set, select the access policy that passes through the most server nodes and has the shortest communication path;

[0039] Delete invalid policies: Delete all access policies that include the server node selected in the previous step from the access policy set;

[0040] Check coverage: Check whether the server nodes in all current access policies cover all server nodes; if so, end this stage and output the results; if not, reselect the access policy;

[0041] Iterative selection: Repeat the above process until all server nodes have been visited;

[0042] Output results: The final output is the selected optimal access set, including the order information of the accessed server nodes, as well as the corresponding communication path and resource consumption.

[0043] Furthermore, the Map and Reduce phases include the following steps:

[0044] Core object selection:

[0045] Each Map task reads data points from a partition and stores them in the form of feature vectors. The improved Cuckoo algorithm is used to process each data point, exploring the data space and identifying the nest location of each data point by generating random numbers.

[0046] Calculate the objective function value for each data point, which includes:

[0047] Density: Calculates the number of data points in the neighborhood around a data point;

[0048] Distance: Calculate the Euclidean distance between a data point and other data points;

[0049] Core object selection criteria: According to the calculated objective function value, the data points with the highest density or the best distance characteristics are selected as core objects as the basis for expanding the cluster;

[0050] Cluster expansion:

[0051] For each core object, the density reachability principle of the DBSCAN algorithm is used to check the data points in its neighborhood, calculate the number of data points in the neighborhood, and use a preset threshold for judgment;

[0052] If the number of data points in the neighborhood exceeds the set threshold, these points are classified into the cluster to which the core object belongs; for each newly added point, the neighborhood check process is repeated and the cluster is continued to expand until no new points are added;

[0053] Clustering results merge:

[0054] In the Reduce task, the clustering results of all Map tasks are collected, and each Map task outputs its own cluster and its member data points. The clusters from different Map tasks are checked for overlap. If there are common data points, they are merged into one cluster.

[0055] Use distance metrics to determine the attribution of data points, ensuring that data points can be correctly classified into corresponding clusters, and finally generate a global clustering structure that includes the clustering results of all Map tasks;

[0056] Noise point processing:

[0057] In the merged clustering results, noise points are identified and processed using an iterative strategy. By re-evaluating the neighborhood of the noise points, they are classified into other clusters. If some noise points find new neighboring data points in subsequent iterations, they are reclassified as cluster members.

[0058] Regularly check and remove noise points in the clustering results, and dynamically adjust the noise point processing strategy by setting thresholds and iteration times.

[0059] Furthermore, the clustering result evaluation includes the following steps:

[0060] S51. Clustering accuracy evaluation: Evaluate the quality of clustering results by calculating the silhouette coefficient:

[0061] For each data point , calculate the average distance between it and other points in the same cluster d ( ), calculate the average distance between the point and the points in the nearest neighbor cluster k ( ), silhouette coefficient s ( ) is calculated as:

[0062] ;

[0063] S52. Execution time evaluation: Record the start and end time of the Map phase and the Reduce phase, calculate the execution time of each phase separately, and get the final total execution time;

[0064] S53, Load Deviation Analysis: Calculate the amount of data processed by each Map task and Reduce task, calculate the average value of the task load, and then calculate the standard deviation of the load deviation LD :

[0065] ;

[0066] Where, D ( h ) is the h The load of the task, n is the total number of tasks, is the average load of all tasks;

[0067] S54, Verify the effectiveness of partitioning strategy and dynamic task allocation: Check the amount of data read by each Map task and the execution time of each task to judge the uniformity of the data partitioning strategy;

[0068] After checking the task allocation, the effectiveness of dynamic task allocation is judged by observing the differences in execution time of different tasks.

[0069] A second aspect of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: adopt the method described above.

[0070] A third aspect of the present disclosure provides a computer device, comprising: a processor and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to: perform the method described above.

[0071] The beneficial effects of the present invention are:

[0072] The present invention first supports various types of data processing based on a distributed computing model. It pre-processes the collected IoT data, uses the KNN algorithm to remove noise data, and uses singular value decomposition to reduce the dimension of the data, thereby reducing the execution time of data processing. Then, a two-stage partitioning strategy is used for the processed data to ensure that the amount of data processed by each Reducer is as uniform as possible, thereby solving the problem of uneven load, dynamically adjusting the partition according to the actual load capacity, and optimizing resource utilization. Through fine-grained partitioning and avoiding additional merge operations, the performance loss caused by data imbalance is reduced, and the overall processing efficiency is improved. Then, by adopting a cluster analysis method, other server nodes with common time window intersections of the current distributed processing server node are found for task allocation, thereby improving data processing efficiency and resource utilization, and optimizing from two aspects: data partitioning and node selection. Finally, a cuckoo algorithm is adopted in the Map and Reduce stages. By attenuating the weight of the probability of finding a nest, the algorithm convergence speed is improved as the number of iterative searches increases, solving the problem that the clustering results are subject to local optimality, thereby improving the accuracy and reliability of clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] To facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0074] Figure 1 A flow chart of a data processing method based on a distributed computing model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0075] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0076] MapReduce, as a widely used distributed computing framework, provides a powerful tool for processing massive amounts of data. The overall architecture of MapReduce includes three parts: input, block, and output. As a software architecture platform for parallel computing and operation, it provides powerful computing capabilities, and can complete the decomposition and processing of tasks in a relatively short period of time, and automatically calculate and distribute tasks. The algorithm integrates parallel computing such as distributed storage, communication, and fault-tolerant processing of data into one platform, and processes complex information through the concept of block, which greatly reduces the hardware requirements of massive data and thus improves computing efficiency. Therefore, the distributed computing model of the present invention uses MapReduce for data clustering, the purpose of which is to provide a basis for subsequently discovering valuable or common data information and models from massive data.

[0077] Specifically, the data processing method and device based on the distributed computing model of the present invention are described in detail through the following content:

[0078] This embodiment provides a data processing method based on a distributed computing model, such as Figure 1 As shown, the following steps are included:

[0079] S1. Data collection and preprocessing: Collect various forms of data from the Internet of Things, including structured, semi-structured, and unstructured data, and preprocess the collected data, including the following steps:

[0080] S11. Data cleaning and transformation: Identify missing values in the data and fill them in using the mean; identify and delete duplicate records; use statistical methods (such as Z-score or IQR) to identify and handle outliers to ensure data quality and reliability;

[0081] Standardize numerical data (such as Z-score standardization or Min-Max scaling) to make all features at the same scale to prevent features from dominating the clustering results; convert categorical variables (such as gender and region) into numerical data (such as one-hot encoding) for MapReduce processing.

[0082] S12. Cleaning Noisy Data: Cross-validation (e.g., k-fold cross-validation) is often used to select the optimal K value. Cross-validation splits the dataset and evaluates the performance of the model under each K value to select the optimal K value to reduce overfitting or underfitting.

[0083] For each data point in the data set, calculate its distance to other data points. The distance calculation methods include:

[0084] Euclidean distance: Applicable to numerical data, measuring the straight-line distance between two data points;

[0085] Manhattan distance: measures the sum of the differences between two data points in each dimension;

[0086] Cosine similarity: used for text data or high-dimensional sparse data, calculating the angle between two vectors.

[0087] For each data point, identify its K nearest neighbors, check the distribution of each category label in the K neighbors or calculate the average value of the K neighbors. If the similarity between a data point and its neighbors is significantly lower than that of most data points, or the frequency of the category label of the point in the neighbors is significantly low, then the data point is judged to be noise.

[0088] The noise data identified by the KNN algorithm is removed from the dataset. Through this process, the quality of the dataset is improved, thereby reducing the impact of abnormal data on subsequent analysis and modeling.

[0089] S13. Reduce the dimensionality of data:

[0090] Constructing a data matrix: Represent the original dataset as a matrix, where rows typically represent samples (data points) and columns represent features (the eigenvalues of each data point). For example, if there are n samples and each sample has m features, the matrix size of the dataset is n×m. If the data is sparse (such as text data), a sparse matrix can be constructed.

[0091] Perform SVD decomposition: Perform singular value decomposition (SVD) on the data matrix. SVD decomposes the original matrix A into three matrices:

[0092] ;

[0093] in, U It is an n×n orthogonal matrix containing the eigenvectors of the samples; S is a diagonal matrix (size is n×m), the elements on its diagonal are singular values, which are the most important features in the data and can represent the variance of the data; It is an m×m orthogonal matrix containing the feature weights or eigenvectors.

[0094] Select principal components: Based on the size of the singular values, select the first K principal components (features). These principal components can capture the most important changes in the data and usually account for most of the variance.

[0095] Constructing a reduced-dimensional dataset: Reconstructing the dataset using the first K selected principal components. By multiplying the first K parts of the data matrix U, S, and V, a new low-dimensional matrix is generated. The original n×m matrix is reduced to n×K (where K is the number of selected principal components). This reduced-dimensional dataset retains the most important information and reduces redundant features.

[0096] It is understood that the data collected in this embodiment is collected from various IoT devices and sensors. Data types include temperature, humidity, location, user behavior, and other information, covering various formats, including structured (such as database table data), semi-structured (such as JSON and XML formatted data), and unstructured (such as text and image data). The K-nearest neighbor algorithm (KNN) is an instance-based learning method that performs classification or regression by calculating the distance between data points. When removing noisy data, the core idea of KNN is that if a data point is significantly less similar to its neighbors than to other data points, or if its neighbors have significant inconsistency in a certain category, then the data point is considered noise. Singular value decomposition (SVD) is a powerful matrix decomposition technique widely used in fields such as data dimensionality reduction and recommender systems. It decomposes the original data matrix into three matrices, extracting the most important features from the data while retaining the low-dimensional information that best represents the data.

[0097] It should be noted that, in this embodiment, for data preprocessing, the K-nearest neighbor technology is used to identify and remove noise data by calculating the distance between the data point and its neighbors, which can effectively improve the data quality; by using singular value decomposition to extract the principal components in the data, the dimension of the data is reduced while retaining the most important information, thereby reducing the computational complexity and improving the subsequent clustering effect.

[0098] S2. Data partitioning and zoning: Divide the data into multiple subsets using a hash algorithm, identify abnormal partitions, and further refine the abnormal partitions using an offset partitioning strategy. This includes the following steps:

[0099] S21, first stage hash partitioning:

[0100] Use a hash algorithm for initial partitioning: Use a traditional hash algorithm (such as MD5, SHA, etc.) to process the input data and map the data to R partitions (R is the number of Reducers) based on the key value. Each partition corresponds to a Reducer, and the data is distributed to different partitions.

[0101] Calculate the data load of each Reducer: Set a data counter on each Mapper node, track the data size in each partition, and calculate the data load received by each Reducer L r :

[0102] ;

[0103] Where, x ij Indicates the i Mapper j Whether the partition is pulled (0 or 1),p ij Indicates the size of the partition; M It represents the number of Mappers, that is, the total number of Mapper nodes involved in data processing; P represents the number of types of partitions contained in each Mapper, that is, the number of different partitions generated by each Mapper.

[0104] Determine abnormal partitions: Set a threshold, usually 90% of the average data volume. r The amount of data for each Reduce Lr If the value exceeds 90% of the threshold, it is considered an abnormal partition. Otherwise, it can be pulled normally by the Reducer for calculation and processing.

[0105] S22, second stage hash partitioning:

[0106] For the abnormal partitions identified in the first stage, we combine the ideas of offset partitioning and fine-grained partitioning to subdivide each abnormal partition into b groups (b is the subdivision multiple);

[0107] During the segmentation process, ensure that data with the same key value is divided into the same group so that subsequent Reduce operations do not require additional merging. Use a new hash function or modify the existing hash function to achieve segmentation and ensure that data is evenly distributed.

[0108] Calculate the current load capacity of each Reducer, evaluate its processing capabilities, and use a greedy algorithm to select the Reducer with the best current load capacity for each segment group to ensure load balancing;

[0109] Calculate the offset between the current optimal Reducer and the original partition number of the group P f ( key ), send the offset value to the Partitioner function, dynamically modify the hash partition function, and the modified Partioner function is:

[0110] ;

[0111] Where, is the modified Partitioner function, which means that after dynamic adjustment, given key The final partition number corresponding to the value; The original Partitioner function indicates the initial partition number corresponding to a given key value when no dynamic adjustment is performed.

[0112] S23, merge partition results:

[0113] Merge the partitioning results of the first phase with the partitioning results of the second phase to form the final partition file. The partition file contains the final partition information corresponding to each key value to ensure that the data can be correctly distributed;

[0114] According to the final partition file, the overall data is directly assigned to each Reducer for processing, ensuring that the data flows to the correct Reducer and optimizing the data processing process.

[0115] It should be noted that the two-stage partitioning strategy detailed in the above steps ensures that the amount of data processed by each Reducer is as even as possible, thereby solving the problem of uneven load. In the second stage, the partitions are dynamically adjusted according to the actual load capacity to optimize resource utilization. Through fine-grained partitioning and avoiding additional merge operations, the performance loss caused by data imbalance is reduced and the overall processing efficiency is improved.

[0116] S3. Dynamic Task Allocation: Server nodes with a common time window are identified through cluster analysis, multiple access policy sets are generated, and the optimal access set that covers all server nodes is selected from the generated access policy sets for task allocation. This includes the following steps:

[0117] Establishing an original set: Establishing an original set that includes all distributed processing server nodes;

[0118] Baseline node selection: The first server node in the set is used as the base, compared with other nodes, and all other server nodes that have a common time window with the base node are selected to form a new set;

[0119] Intersection calculation: among the selected server nodes, continue to calculate the intersection of the time windows, gradually expanding the set until all server nodes related to the benchmark node are found;

[0120] Set update: After completing the processing of a benchmark node, remove the node from the original set, continue to select the next node as the new benchmark, and repeat the above process until all nodes are processed to generate the final multiple access policy sets.

[0121] It should be noted that through this process, the server node access policy set finally generated will include server nodes that can be accessed within the same time window, ensuring that these nodes can be effectively utilized during task scheduling. The purpose of doing this is to optimize the allocation and processing of tasks and improve the overall efficiency and real-time performance of the system.

[0122] Select an access policy set: From the generated access policy set, select the access policy that passes through the most server nodes and has the shortest communication path;

[0123] This means that when choosing, priority should be given to strategies that can cover more server nodes while ensuring the shortest communication path.

[0124] Delete invalid policies: Delete all access policies that include the server node selected in the previous step from the access policy set;

[0125] This is to avoid visiting the same node repeatedly and ensure that each node is visited only once.

[0126] Check coverage: Check whether the server nodes in all current access policies cover all server nodes; if so, end this stage and output the results; if not, reselect the access policy.

[0127] Iterative selection: Repeat the above process until all server nodes have been visited. Each selection must ensure that the selected strategy can further cover the unvisited nodes and maintain the optimal utilization of communication resources.

[0128] Output results: The final output is the selected optimal access set, including the order information of the accessed server nodes, as well as the corresponding communication path and resource consumption.

[0129] It should be noted that the optimal access set refers to a group of server nodes selected from all possible access strategies. The selection of these nodes can achieve the optimization of the objective function (such as minimizing the communication path, maximizing resource utilization, etc.) under the premise of satisfying specific constraints (such as time windows, resource consumption, etc.). By selecting the optimal access set, the efficiency and security of information transmission can be ensured, while reducing the system's resource consumption and time delay.

[0130] S4, Map and Reduce phase: Use the Cuckoo algorithm to select core objects, expand clusters using a density-based clustering algorithm, merge the clustering results of all Map tasks, and use an iterative noise point processing strategy. This includes the following steps:

[0131] Map phase:

[0132] S41. Core object selection:

[0133] Each Map task reads data points from a partition and stores the data points in the form of feature vectors;

[0134] Each data point is processed using the improved cuckoo algorithm (MCS) to explore the data space and identify the nest location of each data point by generating random numbers.

[0135] Among them, the nest position update formula is:

[0136] ;

[0137] Where, Indicates the c The nest in t The position information of the iteration, rand is a random number in the interval [0,1]. and They respectively indicate the locations of the other selected nests.

[0138] Will rand Generated random numbers r With the preset probability of discovery P a For comparison, when r > P a When , the nest location needs to be updated; when r < P a The nest position will no longer be updated iteratively;

[0139] The calculation formula for the discovery probability of the current iteration is:

[0140] ;

[0141] Where, P max and P min They represent the upper and lower limits of the nest discovery probability, G current Indicates the current iteration round, G The preset maximum number of iterations.

[0142] Calculate the objective function value for each data point, which includes:

[0143] Density: Calculates the number of data points in the neighborhood around a data point;

[0144] Distance: Calculates the Euclidean distance between a data point and other data points.

[0145] Selection criteria for core objects: According to the calculated objective function value, the data points with the highest density or the best distance characteristics are selected as core objects as the basis for expanding the clustering cluster.

[0146] S42. Cluster expansion:

[0147] For each core object, the density reachability principle of the DBSCAN algorithm is used to check the data points in its neighborhood and calculate the number of data points in the neighborhood, usually using a preset threshold (such as MinPts) for judgment.

[0148] If the number of data points in the neighborhood exceeds the set threshold, these points will be classified into the cluster to which the core object belongs; for each newly added point, the neighborhood check process is repeated and the cluster is continued to expand until no new points can be added.

[0149] Reduce phase

[0150] S43. Clustering result merging:

[0151] In the Reduce task, the clustering results of all Map tasks are collected, and each Map task outputs its own cluster and its member data points.

[0152] Check whether the clusters from different Map tasks overlap. If there are common data points, merge them into one cluster.

[0153] Deal with the boundary problem of clusters and use distance metrics to determine the attribution of data points to ensure that data points can be correctly classified into the corresponding clusters.

[0154] Finally, a global clustering structure is generated, which contains the clustering results of all Map tasks to ensure the integrity and consistency of the clustering.

[0155] S44, noise point processing:

[0156] In the merged clustering results, noise points are identified.

[0157] It can be understood that noise points refer to data points that do not belong to any cluster, usually points that are excluded because the number of data points in their neighborhood is lower than a set threshold.

[0158] An iterative strategy is used to process the identified noise points. By re-evaluating the neighborhood of the noise points, it is determined whether they can be classified into other clusters. If some noise points find new neighborhood data points in subsequent iterations, they are reclassified as cluster members.

[0159] Regularly check and remove noise points in the clustering results to ensure the accuracy of the clustering results. By setting thresholds and iteration times, dynamically adjust the noise point processing strategy to improve the overall quality of clustering.

[0160] It is understandable that the density-based spatial dutering of applications with noise (DBSCAN) algorithm is primarily used to discover clustering structures in spatial data. Unlike traditional distance-based clustering methods (such as K-means), DBSCAN does not require the number of clusters to be specified in advance, but instead automatically discovers clusters based on the density of data points. In this embodiment, the MCS-KDBSCAN algorithm can effectively implement parallel density clustering within the MapReduce framework, ensuring clustering accuracy and efficiency while also addressing noise issues in large-scale datasets. Meticulous operations at each stage ensure the final clustering results.

[0161] S5. Clustering result evaluation: Use evaluation indicators to evaluate clustering accuracy, analyze job execution time and load deviation, and verify the effectiveness of partitioning strategy and dynamic task allocation. This includes the following steps:

[0162] S51. Clustering accuracy evaluation: Evaluate the quality of clustering results by calculating the silhouette coefficient:

[0163] For each data point , calculate the average distance between it and other points in the same cluster d ( ), calculate the average distance between the point and the points in the nearest neighbor cluster k ( ), silhouette coefficient s ( ) is calculated as:

[0164] ;

[0165] The overall silhouette coefficient of the cluster is the average of the silhouette coefficients of all data points. The value range of the silhouette coefficient is [-1, 1]. The closer it is to 1, the better the clustering quality is, and the closer it is to -1, the poorer the clustering result is.

[0166] S52. Execution time evaluation: Record the start and end time of the Map phase and the Reduce phase, calculate the execution time of each phase separately, and get the final total execution time;

[0167] S53, Load Deviation Analysis: Calculate the amount of data processed by each Map task and Reduce task, calculate the average value of the task load, and then calculate the standard deviation of the load deviation LD :

[0168] ;

[0169] Where, D ( h) is the h The workload of each task represents the computing or processing resources required for each task. n is the total number of tasks (including all Map and Reduce tasks), is the average load of all tasks.

[0170] S54. Verify the effectiveness of partitioning strategy and dynamic task allocation: Check whether the amount of data read by each Map task is close. If the amount of data in some partitions is much higher than that in other partitions, it indicates that the partitioning strategy is uneven. Observe whether the execution time of each task is close. If some Map or Reduce tasks are significantly different from other tasks, it indicates that the data is uneven or the partitioning is unreasonable.

[0171] After task allocation, check whether the load on each node is balanced. Also, observe the difference in execution time between different tasks. If dynamic task allocation is effective, the difference in task execution time should be small.

[0172] S6. Visualization: Choose appropriate visualization tools to display the clustering results, and design appropriate chart types (such as scatter plots, heat maps, dendrograms, etc.) based on the clustering results to clearly display the distribution, characteristics, and associations of the clusters.

[0173] The clustering results are displayed graphically, highlighting the core clusters, noise data, and the relationship between data points to ensure easy understanding and analysis.

[0174] This embodiment further provides a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: adopt a data processing method based on a distributed computing model as described above.

[0175] This embodiment also provides a computer device, which includes: a processor; and a memory arranged to store computer-executable instructions, which, when executed, enable the processor to: a data processing method based on a distributed computing model as described above.

[0176] The present invention first supports various types of data processing based on a distributed computing model. It pre-processes the collected IoT data, uses the KNN algorithm to remove noise data, and uses singular value decomposition to reduce the dimension of the data, thereby reducing the execution time of data processing. Then, a two-stage partitioning strategy is used for the processed data to ensure that the amount of data processed by each Reducer is as uniform as possible, thereby solving the problem of uneven load, dynamically adjusting the partition according to the actual load capacity, and optimizing resource utilization. Through fine-grained partitioning and avoiding additional merge operations, the performance loss caused by data imbalance is reduced, and the overall processing efficiency is improved. Then, by adopting a cluster analysis method, other server nodes with common time window intersections of the current distributed processing server node are found for task allocation, thereby improving data processing efficiency and resource utilization, and optimizing from two aspects: data partitioning and node selection. Finally, a cuckoo algorithm is adopted in the Map and Reduce stages. By attenuating the weight of the probability of finding a nest, the algorithm convergence speed is improved as the number of iterative searches increases, solving the problem that the clustering results are subject to local optimality, thereby improving the accuracy and reliability of clustering.

[0177] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as above in terms of a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can, without departing from the scope of the technical solution of the present invention, make some changes or modifications to equivalent embodiments using the technical contents disclosed above. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A data processing method based on a distributed computing model, characterized in that: The following steps are involved: Data collection and preprocessing: Collect various forms of data from the Internet of Things and preprocess the collected data, including data cleaning and conversion, removing noisy data, and reducing data dimensionality; Data partitioning and zoning: Use a hash algorithm to perform preliminary data partitioning, calculate the load of each Reducer, identify and adjust abnormal partitions, refine abnormal partitions through fine-grained partitioning, and dynamically adjust the partitioning strategy; Dynamic task allocation: Based on the time window of the server node, identify the nodes with the same time window and generate an access policy set, and select the optimal access set for task allocation; Map and Reduce phase: Select core objects, expand clusters using a density-based clustering algorithm, merge the clustering results of all map tasks, and use an iterative noise point processing strategy; Clustering result evaluation: Use evaluation metrics to assess clustering accuracy, analyze job execution time and load deviation, and verify the effectiveness of partitioning strategies and dynamic task allocation; Visualization: Select appropriate visualization tools to display clustering results, design chart types based on clustering results, and display clustering results in a graphical way.

2. The data processing method based on a distributed computing model according to claim 1, characterized in that: The data collection and preprocessing includes the following steps: Data cleaning and conversion: Identify missing values in the data and fill them by taking the mean; identify and delete duplicate records, use statistical methods to identify and handle outliers, standardize numerical data, and convert categorical variables to numerical data; Clean up noisy data: select the best K value through cross-validation. For each data point in the dataset, calculate its distance to other data points, and for each data point, identify its K nearest neighbors. Check the distribution of each category label in the K neighbors or calculate the average value of the K neighbors. If the similarity between a data point and its neighbors is significantly lower than that of most data points, or the frequency of the category label of the point in the neighbors is significantly lower, then the data point is judged as noise and removed. Reduce the dimensionality of the data: Represent the original data set as a matrix, perform singular value decomposition on the data matrix, select the first K principal components based on the size of the singular values, and reconstruct the data set using the selected first K principal components.

3. The data processing method based on a distributed computing model according to claim 1, characterized in that: The data division and partitioning includes the following steps: First stage hash partitioning: Use the traditional hash algorithm to process the input data and map the data to R partitions based on the key value. Each partition corresponds to a Reducer, and the data is distributed to different partitions. Set up a data counter on each Mapper node to track the size of the data in each partition and calculate the data load received by each Reducer L r : ； Where, x ij Indicates the i Mapper j Whether the partition is pulled, p ij Indicates the size of the partition; M Indicates the number of Mappers, and P indicates the number of partition types contained in each Mapper; Set a threshold value. r The amount of data for each Reduce Lr If the value exceeds 90% of the threshold, it is considered an abnormal partition. Otherwise, it will be pulled normally by the Reducer for calculation and processing.

4. The data processing method based on a distributed computing model according to claim 3, characterized in that: The data division and partitioning further includes the steps of: Second stage hash partitioning: Based on the abnormal partitions identified in the first phase, each abnormal partition is subdivided into b groups; the current load capacity of each Reducer is calculated, and the Reducer with the best current load capacity is selected for each subdivided group using a greedy algorithm; Calculate the offset between the current optimal Reducer and the original partition number of the group , send the offset value to the Partitioner function, dynamically modify the hash partition function, and the modified Partioner function is: ； Where, is the modified Partitioner function, is the original Partitioner function; Merge partition results: Merge the partition results of the first stage with the partition results of the second stage to form the final partition file. The partition file contains the final partition information corresponding to each key value. Based on the final partition file, the overall data is directly assigned to each Reducer for processing.

5. The data processing method based on a distributed computing model according to claim 1, characterized in that: The dynamic task allocation includes the following steps: Establishing an original set: Establishing an original set that includes all distributed processing server nodes; Baseline node selection: The first server node in the set is used as the base, compared with other nodes, and all other server nodes that have a common time window with the base node are selected to form a new set; Intersection calculation: among the selected server nodes, continue to calculate the intersection of the time windows, gradually expanding the set until all server nodes related to the benchmark node are found; Set update: After completing the processing of a benchmark node, remove the node from the original set, continue to select the next node as the new benchmark, and repeat the above process until all nodes are processed to generate the final multiple access policy sets.

6. The data processing method based on a distributed computing model according to claim 5, characterized in that: The dynamic task allocation further includes the steps of: Select an access policy set: From the generated access policy set, select the access policy that passes through the most server nodes and has the shortest communication path; Delete invalid policies: Delete all access policies that include the server node selected in the previous step from the access policy set; Check coverage: Check whether the server nodes in all current access policies cover all server nodes; If yes, then end this phase and output the result; if not, reselect the access strategy; Iterative selection: Repeat the above process until all server nodes have been visited; Output results: The final output is the selected optimal access set, including the order information of the accessed server nodes, as well as the corresponding communication path and resource consumption.

7. The data processing method based on a distributed computing model according to claim 1, characterized in that: The Map and Reduce phases include the following steps: Core object selection: Each Map task reads data points from a partition and stores them in the form of feature vectors. The improved Cuckoo algorithm is used to process each data point, exploring the data space and identifying the nest location of each data point by generating random numbers. Calculate the objective function value for each data point, which includes: Density: Calculates the number of data points in the neighborhood around a data point; Distance: Calculate the Euclidean distance between a data point and other data points; Core object selection criteria: According to the calculated objective function value, the data points with the highest density or the best distance characteristics are selected as core objects as the basis for expanding the cluster; Cluster expansion: For each core object, the density reachability principle of the DBSCAN algorithm is used to check the data points in its neighborhood, calculate the number of data points in the neighborhood, and use a preset threshold for judgment; If the number of data points in the neighborhood exceeds the set threshold, these points are classified into the cluster to which the core object belongs; for each newly added point, the neighborhood check process is repeated and the cluster is continued to expand until no new points are added; Clustering results merge: In the Reduce task, the clustering results of all Map tasks are collected, and each Map task outputs its own cluster and its member data points. The clusters from different Map tasks are checked for overlap. If there are common data points, they are merged into one cluster. Use distance metrics to determine the attribution of data points, ensuring that data points can be correctly classified into corresponding clusters, and finally generate a global clustering structure that includes the clustering results of all Map tasks; Noise point processing: In the merged clustering results, noise points are identified and processed using an iterative strategy. By re-evaluating the neighborhood of the noise points, they are classified into other clusters. If some noise points find new neighboring data points in subsequent iterations, they are reclassified as cluster members. Regularly check and remove noise points in the clustering results, and dynamically adjust the noise point processing strategy by setting thresholds and iteration times.

8. The data processing method based on a distributed computing model according to claim 1, characterized in that: The clustering result evaluation includes the following steps: Clustering accuracy evaluation: The quality of clustering results is evaluated by calculating the silhouette coefficient: For each data point , calculate the average distance between it and other points in the same cluster d ( ), calculate the average distance between the point and the points in the nearest neighbor cluster k ( ), silhouette coefficient s ( ) is calculated as: ； Execution time evaluation: Record the start and end times of the Map and Reduce phases, calculate the execution time of each phase separately, and get the final total execution time; Load deviation analysis: Calculate the amount of data processed by each Map task and Reduce task, calculate the average value of the task load, and then calculate the standard deviation of the load deviation LD : ； Where, D ( h ) is the h The load of the task, n is the total number of tasks, is the average load of all tasks; Verify the effectiveness of partitioning strategy and dynamic task allocation: Check the amount of data read by each Map task and the execution time of each task to determine the uniformity of the data partitioning strategy; After checking the task allocation, the effectiveness of dynamic task allocation is judged by observing the differences in execution time of different tasks.

9. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer executable instructions are executed by a processor, the processor is caused to: adopt the method according to any one of claims 1 to 8.

10. A computer device, wherein: include: A processor and a memory arranged to store computer executable instructions, which when executed cause the processor to: perform the method according to any one of claims 1 to 8.

Images