Data processing method and equipment based on distributed computing model

Abstract

The invention relates to a data processing method and equipment based on a distributed computing model, and belongs to the technical field of data processing. Comprising the following steps: performing preliminary partitioning on data by using a Hash algorithm, identifying and adjusting abnormal partitions, and refining the abnormal partitions through fine-grained partitioning; nodes with the same time window are identified, an access strategy set is generated, and an optimal access set is selected for task allocation; and selecting a core object, expanding a cluster through a density-based clustering algorithm, and merging clustering results of all Map tasks. According to the method, through a two-stage division strategy, the uniformity of the data size processed by each Reducer is improved, and the optimal access set of the current distributed processing server nodes is found out for task allocation by adopting a clustering analysis method, so that the data processing efficiency and the resource utilization rate are improved, and the resource utilization rate is increased. And finally, noise points are removed through a density-based clustering algorithm, so that the accuracy of a clustering result is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of data processing, and in particular relates to a data processing method and device based on a distributed computing model. Background Art

[0002] With the rapid growth of data in the era of big data, the traditional single-machine computing model can no longer meet the processing needs. Therefore, data processing methods based on distributed computing models have emerged. Especially when complex calculations such as cluster analysis are required for large amounts of data, distributed computing can effectively improve computing efficiency.

[0003] However, data processing methods based on distributed computing still face some challenges in practical applications: in a distributed environment, improper data division may cause some nodes to be overloaded, affecting the overall computing efficiency; in the clustering process, uneven data distribution may cause some nodes to have too heavy processing tasks; at the same time, noise points and outliers will affect the clustering results. Summary of the invention

[0004] In order to solve the above-mentioned problems existing in the prior art, the present invention provides a data processing method and device based on a distributed computing model. Through a two-stage partitioning strategy, the uniformity of the amount of data processed by each Reducer is improved, and then by adopting a clustering analysis method, the optimal access set of the current distributed processing server nodes is found for task allocation, thereby improving data processing efficiency and resource utilization. Finally, noise points are removed through a density-based clustering algorithm to improve the accuracy of the clustering results.

[0005] The purpose of the present invention can be achieved through the following technical solutions: A first aspect of the present disclosure provides a data processing method based on a distributed computing model, comprising the following steps: S1. Data collection and preprocessing: Collect various forms of data from the Internet of Things and preprocess the collected data, including data cleaning and conversion, removing noise data and reducing the dimension of data; S2. Data division and partitioning: Use hash algorithms to perform preliminary data partitioning, calculate the load of each Reducer, identify and adjust abnormal partitions, refine abnormal partitions through fine-grained partitioning, and dynamically adjust partitioning strategies; S3, dynamic task allocation: according to the time window of the server node, identify the nodes with the same time window and generate an access policy set, and select the optimal access set for task allocation; S4, Map and Reduce phase: Select core objects, expand clusters through density-based clustering algorithms, merge clustering results of all Map tasks, and use iterative noise point processing strategies; S5. Clustering result evaluation: Use evaluation indicators to evaluate clustering accuracy, analyze job execution time and load deviation, and verify the effectiveness of partitioning strategy and dynamic task allocation; S6. Visualization: Select appropriate visualization tools to display the clustering results, design chart types according to the clustering results, and display the clustering results in a graphical way.

[0006] Furthermore, the data collection and preprocessing includes the following steps: S11. Data cleaning and conversion: Identify missing values ​​in the data and fill them in by taking the mean; identify and delete duplicate records, use statistical methods to identify and process outliers, standardize numerical data, and convert categorical variables into numerical data; S12. Clear noise data: Select the best K value through cross-validation. For each data point in the data set, calculate its distance to other data points, and for each data point, identify its K nearest neighbors. Check the distribution of each category label in the K neighbors or calculate the average of the values ​​of the K neighbors. If the similarity between a data point and its neighbors is significantly lower than that of most data points, or the frequency of the category label of the point in the neighbors is significantly lower, the data point is judged as noise and removed; S13. Reduce the dimensionality of the data: Represent the original data set as a matrix, perform singular value decomposition on the data matrix, select the first K principal components according to the size of the singular values, and use the selected first K principal components to reconstruct the data set.

[0007] Furthermore, the data division and partitioning includes the following steps: S21, first stage hash partitioning: Use the traditional hash algorithm to process the input data and map the data to R partitions according to the key value. Each partition corresponds to a Reducer, and the data is distributed to different partitions. Set up data counters on each Mapper node to track the size of the data in each partition and calculate the data load received by each Reducer L r : ; In the formula, x ij Indicates i Mapper j Whether the partition is pulled, p ij Indicates the size of the partition; M Indicates the number of Mappers, and P indicates the number of partition types contained in each Mapper; Set a threshold. r The amount of data for each Reduce Lr If the value exceeds 90% of the threshold, it is considered an abnormal partition. Otherwise, it is pulled normally by the Reducer for calculation and processing.

[0008] Furthermore, the data division and partitioning also includes the steps of: S22, second stage hash partitioning: According to the abnormal partitions identified in the first stage, each abnormal partition is subdivided into b groups; the current load capacity of each Reducer is calculated, and the Reducer with the best current load capacity is selected for each subdivided group using a greedy algorithm; Calculate the offset between the current optimal Reducer and the original partition number of the group , send the offset value to the Partitioner function, dynamically modify the hash partition function, and the modified Partioner function is: ; In the formula, is the modified Partitioner function, is the original Partitioner function; S23. Merge partition results: Merge the partition results of the first stage with the partition results of the second stage to form a final partition file. The partition file contains the final partition information corresponding to each key value. According to the final partition file, the overall data is directly allocated to each Reducer for processing.

[0009] Furthermore, the dynamic task allocation includes the following steps: Establishing the original set: Establishing an original set that includes all distributed processing server nodes; Benchmark node selection: Take the first server node in the set as the benchmark, compare it with other nodes, and select all other server nodes that have a common time window with the benchmark node to form a new set; Intersection calculation: among the selected server nodes, continue to calculate the intersection of the time windows and gradually expand the set until all server nodes related to the benchmark node are found; Set update: After completing the processing of a benchmark node, remove the node from the original set, continue to select the next node as the new benchmark, and repeat the above process until all nodes are processed to generate the final multiple access policy sets.

[0010] Furthermore, the dynamic task allocation further includes the steps of: Select an access policy set: From the generated access policy set, select the access policy that passes through the most server nodes and has the shortest communication line; Delete invalid policies: Delete all access policies that include the server node selected in the previous step from the access policy set; Check coverage: Check whether the server nodes in all current access policies cover all server nodes; if yes, end this stage and output the results; if not, reselect the access policy; Iterative selection: Repeat the above process until all server nodes have been visited; Output results: The final output is the selected optimal access set, including the order information of the accessed server nodes, as well as the corresponding communication path and resource consumption.

[0011] Furthermore, the Map and Reduce phases include the following steps: Core object selection: Each Map task reads data points from a partition and stores them in the form of feature vectors. Each data point is processed using the improved cuckoo algorithm, which generates random numbers to explore the data space and identify the nest location of each data point. Calculate the objective function value for each data point, which includes: Density: Calculates the number of data points in the neighborhood around a data point; Distance: Calculate the Euclidean distance between a data point and other data points; Selection criteria for core objects: According to the calculated objective function value, the data points with the highest density or the best distance characteristics are selected as core objects as the basis for expanding the clustering cluster; Cluster expansion: For each core object, use the density reachability principle of the DBSCAN algorithm to check the data points in its neighborhood, calculate the number of data points in the neighborhood, and use a preset threshold to judge; If the number of data points in the neighborhood exceeds the set threshold, these points are classified into the cluster to which the core object belongs; for each newly added point, the neighborhood check process is repeated and the cluster is continued to expand until no new points are added; Clustering results merge: In the Reduce task, the clustering results of all Map tasks are collected, and each Map task outputs its own cluster and its member data points; check whether the clusters from different Map tasks overlap, and if there are common data points, merge them into one cluster; Use distance metrics to determine the attribution of data points, ensure that data points can be correctly classified into corresponding clusters, and finally generate a global clustering structure that contains the clustering results of all Map tasks; Noise point processing: In the merged clustering results, noise points are identified and processed using an iterative strategy. The noise points are classified into other clusters by re-evaluating their neighborhoods. If some noise points find new neighborhood data points in subsequent iterations, they are reclassified as cluster members. Regularly check and remove noise points in the clustering results, and dynamically adjust the noise point processing strategy by setting thresholds and iteration times.

[0012] Furthermore, the clustering result evaluation comprises the following steps: S51. Clustering accuracy evaluation: Evaluate the quality of clustering results by calculating the silhouette coefficient: For each data point , calculate the average distance between it and other points in the same cluster d ( ), calculate the average distance between the point and the points in the nearest neighbor cluster k ( ), silhouette coefficient s ( ) is calculated as: ; S52, execution time evaluation: record the start and end time of the Map phase and the Reduce phase, calculate the execution time of each phase respectively to obtain the final total execution time; S53, Load Deviation Analysis: Calculate the amount of data processed by each Map task and Reduce task, calculate the average value of the task load, and then calculate the standard deviation of the load deviation LD : ; In the formula, D ( h ) is the h The load of the task, n is the total number of tasks, is the average load of all tasks; S54, validation of the partitioning strategy and dynamic task allocation: check the amount of data read by each Map task and the execution time of each task to determine the uniformity of the data partitioning strategy; After checking the task allocation, the effectiveness of dynamic task allocation is judged by observing the differences in execution time of different tasks.

[0013] A second aspect of the present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is caused to: adopt the method as described above.

[0014] A third aspect of the present disclosure provides a computer device, comprising: a processor and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to: perform the method as described above.

[0015] The beneficial effects of the present invention are: The invention firstly supports various types of data processing based on a distributed computing model, pre-processes the collected IoT data, uses the KNN algorithm to remove noise data, uses singular value decomposition to reduce the dimension of the data, and reduces the execution time of data processing; then, a two-stage partitioning strategy is used for the processed data to ensure that the amount of data processed by each Reducer is as uniform as possible, thereby solving the problem of uneven load, dynamically adjusting the partition according to the actual load capacity, and optimizing resource utilization; through fine-grained partitioning and avoiding additional merging operations, the performance loss caused by data imbalance is reduced, and the overall processing efficiency is improved; then, by adopting a clustering analysis method, other server nodes with common time window intersections of the current distributed processing server nodes are found for task allocation, thereby improving data processing efficiency and resource utilization, and optimizing from two aspects of data partitioning and node selection; finally, a cuckoo algorithm is adopted in the Map and Reduce stages, and the convergence speed of the algorithm is improved as the number of iterative searches increases by attenuating the weight of the probability of finding a nest, thereby solving the problem that the clustering result is subject to the local optimum, thereby improving the accuracy and reliability of clustering. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0017] Figure 1 A flow chart of a data processing method based on a distributed computing model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.

[0019] As a widely used distributed computing framework, MapReduce provides a powerful tool for processing massive data. The overall architecture of MapReduce includes three parts: input, block and output. As a software architecture platform for parallel computing and operation, it provides powerful computing power, and can complete the decomposition and processing of tasks in a relatively short time, and automatically calculate and distribute tasks. The algorithm integrates parallel computing such as distributed storage, communication and fault-tolerant processing of data into one platform, and processes complex information through the concept of block, which greatly reduces the hardware requirements of massive data and thus improves computing efficiency. Therefore, the distributed computing model of the present invention uses MapReduce for data clustering, and its purpose is to provide a basis for subsequently discovering valuable or common data information and models from massive data.

[0020] Specifically, a data processing method and device based on a distributed computing model of the present invention are described in detail through the following content: This embodiment provides a data processing method based on a distributed computing model, such as Figure 1 As shown, the following steps are included: S1. Data collection and preprocessing: Collect various forms of data from the Internet of Things, including structured, semi-structured and unstructured data, and preprocess the collected data, including the following steps: S11. Data cleaning and transformation: Identify missing values ​​in the data and fill them in by averaging; identify and delete duplicate records, use statistical methods (such as Z-score or IQR) to identify and process outliers, and ensure the quality and reliability of the data; Standardize the numerical data (such as Z-score standardization or Min-Max scaling) to make all features at the same scale to avoid features dominating the clustering results; convert categorical variables (such as gender, region) to numerical data (such as one-hot encoding) for MapReduce processing.

[0021] S12. Clean up noisy data: Cross-validation (such as k-fold cross-validation) is usually used to select the best K value. Cross-validation selects the best K value to reduce overfitting or underfitting by splitting the data set and evaluating the performance of the model under each K value.

[0022] For each data point in the data set, calculate its distance to other data points. The distance calculation method includes: Euclidean distance: Applicable to numerical data, measuring the straight-line distance between two data points; Manhattan distance: measures the sum of the differences between two data points in each dimension; Cosine similarity: used for text data or high-dimensional sparse data to calculate the angle between two vectors.

[0023] For each data point, identify its K nearest neighbors, check the distribution of each category label in the K neighbors or calculate the average value of the K neighbors. If the similarity between a data point and its neighbors is significantly lower than that of most data points, or the frequency of the category label of the point in the neighbors is significantly low, the data point is judged to be noise.

[0024] The noise data identified by the KNN algorithm is removed from the dataset. Through this process, the quality of the dataset is improved, thereby reducing the impact of abnormal data on subsequent analysis and modeling.

[0025] S13. Reduce the dimensionality of data: Construct a data matrix: Represent the original data set as a matrix, where rows usually represent samples (data points) and columns represent features (feature values ​​of each data point). For example, suppose there are n samples, each sample has m features, and the matrix size of the data set is n×m; if the data is sparse (such as text data), a sparse matrix can be constructed.

[0026] Perform SVD decomposition: Perform singular value decomposition (SVD) on the data matrix. SVD decomposes the original matrix A into three matrices: ; in, U is an n×n orthogonal matrix containing the eigenvectors of the samples; S is a diagonal matrix (size is n×m), the elements on its diagonal are singular values, which are the most important features in the data and can represent the variance of the data; It is an m×m orthogonal matrix containing the feature weights or eigenvectors.

[0027] Select principal components: Based on the size of the singular values, select the first K principal components (features). These principal components can capture the most important changes in the data and usually account for most of the variance.

[0028] Construct the reduced-dimensionality dataset: Reconstruct the dataset using the selected first K principal components. By multiplying the first K parts of U, S, and V in the data matrix, a new low-dimensional matrix is ​​obtained, and the dimension of the matrix changes from the original n×m to n×K (where K is the number of selected principal components). The reduced-dimensional dataset retains the most important information in the data and reduces redundant features.

[0029] It is understandable that the data collected in this embodiment is collected from various IoT devices and sensors. The data types include information such as temperature, humidity, location, user behavior, etc., covering various forms such as structured (such as database table data), semi-structured (such as JSON, XML format data) and unstructured (such as text, image data). K-nearest neighbor algorithm (KNN) is an instance-based learning method that performs classification or regression by calculating the distance between data points. When removing noisy data, the core idea of ​​KNN is that if a data point has a significantly lower similarity to its neighbors than other data points, or its neighbors have obvious inconsistencies in a certain category, the data point is regarded as noise. Singular value decomposition (SVD) is a powerful matrix decomposition technology that is widely used in data dimensionality reduction, recommendation systems and other fields. It extracts the most important features of the data by decomposing the original data matrix into three matrices, and retains the low-dimensional information that best represents the data.

[0030] It should be noted that in this embodiment, for data preprocessing, the K-nearest neighbor technology is used to calculate the distance between the data point and its neighbors, identify and remove noise data, which can effectively improve the data quality; by using singular value decomposition to extract the principal components in the data, the dimension of the data is reduced while retaining the most important information, thereby reducing the computational complexity and improving the subsequent clustering effect.

[0031] S2. Data division and partitioning: Divide the data into multiple subsets through the hash algorithm, identify abnormal partitions, and use the offset partition strategy to further refine the abnormal partitions, including the following steps: S21, first stage hash partitioning: Use hash algorithm for initial partitioning: Use traditional hash algorithm (such as MD5, SHA, etc.) to process input data, map data to R partitions (R is the number of Reducers) according to key value, each partition corresponds to a Reducer, and data is distributed to different partitions; Calculate the data load of each Reducer: Set a data counter on each Mapper node to track the size of the data in each partition and calculate the data load received by each Reducer L r : ; In the formula, x ij Indicates i Mapper j Whether the partition is pulled (0 or 1), p ij Indicates the size of the partition; MIt represents the number of Mappers, that is, the total number of Mapper nodes involved in data processing; P represents the number of types of partitions contained in each Mapper, that is, the number of different partitions generated by each Mapper.

[0032] Determine abnormal partitions: Set a threshold, usually 90% of the average data volume. r The amount of data for each Reduce Lr If the value exceeds 90% of the threshold, it is considered an abnormal partition. Otherwise, it can be pulled normally by the Reducer for calculation and processing.

[0033] S22, second stage hash partitioning: For the abnormal partitions identified in the first stage, each abnormal partition is subdivided into b groups (b is the subdivision multiple) by combining the ideas of offset partitioning and fine-grained partitioning. During the segmentation process, ensure that data with the same key value is divided into the same group so that subsequent Reduce operations do not require additional merging. Use a new hash function or modify the existing hash function to achieve segmentation and ensure that data is evenly distributed. Calculate the current load capacity of each Reducer, evaluate its processing capacity, and use a greedy algorithm to select the Reducer with the best current load capacity for each segment group to ensure load balancing; Calculate the offset between the current optimal Reducer and the original partition number of the group P f ( key ), send the offset value to the Partitioner function, dynamically modify the hash partition function, and the modified Partioner function is: ; In the formula, is the modified Partitioner function, indicating that after dynamic adjustment, given key The final partition number corresponding to the value; It is the original Partitioner function, which indicates the initial partition number corresponding to the given key value when no dynamic adjustment is performed.

[0034] S23, merge partition results: The partition results of the first phase are combined with the partition results of the second phase to form the final partition file. The partition file contains the final partition information corresponding to each key value to ensure that the data can be correctly distributed; According to the final partition file, the overall data is directly distributed to each Reducer for processing, ensuring that the data flows to the correct Reducer and optimizing the data processing process.

[0035] It should be noted that the two-stage partitioning strategy in the above detailed steps ensures that the amount of data processed by each Reducer is as uniform as possible, thereby solving the problem of uneven load. In the second stage, the partitions are dynamically adjusted according to the actual load capacity to optimize resource utilization; through fine-grained partitioning and avoiding additional merge operations, the performance loss caused by data imbalance is reduced and the overall processing efficiency is improved.

[0036] S3, dynamic task allocation: server nodes with common time windows are identified by cluster analysis, multiple access policy sets are generated, and the optimal access set that can cover all server nodes is selected from the generated access policy sets for task allocation, including the following steps: Establishing the original set: Establishing an original set that includes all distributed processing server nodes; Benchmark node selection: Take the first server node in the set as the benchmark, compare it with other nodes, and select all other server nodes that have a common time window with the benchmark node to form a new set; Intersection calculation: among the selected server nodes, continue to calculate the intersection of the time windows and gradually expand the set until all server nodes related to the benchmark node are found; Set update: After completing the processing of a benchmark node, remove the node from the original set, continue to select the next node as the new benchmark, and repeat the above process until all nodes are processed to generate the final multiple access policy sets.

[0037] It should be noted that through this process, the server node access policy set finally generated will include server nodes that can be accessed within the same time window, ensuring that these nodes can be effectively utilized during task scheduling. The purpose of this is to optimize the allocation and processing of tasks and improve the overall efficiency and real-time performance of the system.

[0038] Select an access policy set: From the generated access policy set, select the access policy that passes through the most server nodes and has the shortest communication line; This means that when choosing, priority should be given to strategies that can cover more server nodes while ensuring the shortest communication path.

[0039] Delete invalid policies: Delete all access policies that include the server node selected in the previous step from the access policy set; This is to avoid visiting the same node repeatedly and ensure that each node is visited only once.

[0040] Check coverage: Check whether the server nodes in all current access policies cover all server nodes; if yes, end this stage and output the results; if not, reselect the access policy.

[0041] Iterative selection: Repeat the above process until all server nodes have been visited. Each selection must ensure that the selected strategy can further cover unvisited nodes and maintain optimal utilization of communication resources.

[0042] Output results: The final output is the selected optimal access set, including the order information of the accessed server nodes, as well as the corresponding communication path and resource consumption.

[0043] It should be noted that: the optimal access set refers to a group of server nodes selected from all possible access strategies. The selection of these nodes can achieve the optimization of the objective function (such as minimizing the communication path, maximizing resource utilization, etc.) under the premise of meeting specific constraints (such as time windows, resource consumption, etc.). By selecting the optimal access set, the efficiency and security of information transmission can be ensured, while reducing the system's resource consumption and time delay.

[0044] S4, Map and Reduce phase: Use the cuckoo algorithm to select core objects, expand clustering clusters through density-based clustering algorithm, merge clustering results of all Map tasks, and use iterative noise point processing strategy, including the following steps: Map phase: S41. Core object selection: Each Map task reads data points from a partition, and the data points are stored in the form of feature vectors; Each data point is processed using the modified cuckoo algorithm (MCS) to explore the data space and identify the nest location of each data point by generating random numbers.

[0045] Among them, the nest position update formula is: ; In the formula, Indicates c The nest is in t The position information of the iteration, rand is a random number in the interval [0,1]. and They respectively represent the locations of other selected nests.

[0046] Will rand Random numbers generated r With the preset probability of discovery P a For comparison, when r > P a When , the nest location needs to be updated; when r < P a The nest position will no longer be updated iteratively; The discovery probability calculation formula for the current iteration is: ; In the formula, P max and P min They represent the upper and lower limits of the nest discovery probability, G current Indicates the current iteration round, G The preset maximum number of iterations.

[0047] Calculate the objective function value for each data point, which includes: Density: Calculates the number of data points in the neighborhood around a data point; Distance: Calculates the Euclidean distance between a data point and other data points.

[0048] Selection criteria for core objects: According to the calculated objective function value, the data points with the highest density or the best distance characteristics are selected as core objects as the basis for expanding the clustering cluster.

[0049] S42, Cluster expansion: For each core object, the density reachability principle of the DBSCAN algorithm is used to check the data points in its neighborhood and calculate the number of data points in the neighborhood, usually using a preset threshold (such as MinPts) to make a judgment.

[0050] If the number of data points in the neighborhood exceeds the set threshold, these points are classified into the cluster to which the core object belongs; for each newly added point, the neighborhood check process is repeated and the cluster continues to be expanded until no new points can be added.

[0051] Reduce Phase S43, clustering results merge: In the Reduce task, the clustering results of all Map tasks are collected, and each Map task outputs its own clustering cluster and its member data points.

[0052] Check whether the clusters from different Map tasks overlap. If there are common data points, merge them into one cluster.

[0053] Deal with the boundary problem of clusters and use distance metrics to determine the attribution of data points to ensure that data points can be correctly classified into the corresponding clusters.

[0054] Finally, a global clustering structure is generated, which contains the clustering results of all Map tasks to ensure the integrity and consistency of the clustering.

[0055] S44, noise point processing: In the merged clustering results, noise points are identified.

[0056] It can be understood that noise points refer to data points that do not belong to any cluster, usually points that are excluded because the number of data points in their neighborhood is lower than a set threshold.

[0057] An iterative strategy is used to process the identified noise points. By re-evaluating the neighborhood of the noise points, it is determined whether they can be classified into other clusters. If some noise points find new neighborhood data points in subsequent iterations, they are reclassified as cluster members.

[0058] Regularly check and remove noise points in the clustering results to ensure the accuracy of the clustering results. By setting thresholds and iteration times, dynamically adjust the noise point processing strategy to improve the overall quality of clustering.

[0059] It is understandable that the density-based spatial dutering ofapplications with noise (DBSCAN) algorithm is mainly used to discover clustering structures in spatial data. Unlike traditional distance-based clustering methods (such as K-means), DBSCAN does not require the number of clusters to be specified in advance, but automatically discovers clusters by the density of data points. In this embodiment, the MCS-KDBSCAN algorithm can effectively implement parallel density clustering under the MapReduce framework, ensuring the accuracy and efficiency of clustering, while being able to handle noise problems in large-scale data sets. The meticulous operation of each stage provides a guarantee for the final clustering result.

[0060] S5. Clustering result evaluation: Use evaluation indicators to evaluate clustering accuracy, analyze job execution time and load deviation, and verify the effectiveness of partitioning strategy and dynamic task allocation, including the following steps: S51. Clustering accuracy evaluation: Evaluate the quality of clustering results by calculating the silhouette coefficient: For each data point , calculate the average distance between it and other points in the same cluster d ( ), calculate the average distance between the point and the points in the nearest neighbor cluster k ( ), silhouette coefficient s ( ) is calculated as: ; The overall silhouette coefficient of the cluster is the average of the silhouette coefficients of all data points. The value range of the silhouette coefficient is [-1, 1]. The closer it is to 1, the better the clustering quality is, and the closer it is to -1, the poorer the clustering result is.

[0061] S52, execution time evaluation: record the start and end time of the Map phase and the Reduce phase, calculate the execution time of each phase respectively to obtain the final total execution time; S53, Load Deviation Analysis: Calculate the amount of data processed by each Map task and Reduce task, calculate the average value of the task load, and then calculate the standard deviation of the load deviation LD : ; In the formula, D ( h ) is the h The load of each task represents the computing or processing resources required for each task. n is the total number of tasks (including all Map and Reduce tasks), is the average load of all tasks.

[0062] S54. Verify the effectiveness of partitioning strategy and dynamic task allocation: Check whether the amount of data read by each Map task is close. If the amount of data in some partitions is much higher than that in other partitions, it means that the partitioning strategy is uneven. Observe whether the execution time of each task is close. If some Map or Reduce tasks are significantly different from other tasks, it means that the data is uneven or the partitioning is unreasonable.

[0063] Check whether the load of each node is balanced after task allocation, and observe the difference in execution time of different tasks. If dynamic task allocation is effective, the difference in task execution time should be small.

[0064] S6. Visualization: Choose appropriate visualization tools to display the clustering results, and design appropriate chart types (such as scatter plots, heat maps, dendrograms, etc.) based on the clustering results to clearly display the distribution, characteristics and associations of the clusters.

[0065] The clustering results are displayed graphically, highlighting the core clusters, noise data, and the relationship between data points to ensure easy understanding and analysis.

[0066] This embodiment also provides a computer-readable storage medium having computer-executable instructions stored thereon, wherein when the computer-executable instructions are executed by a processor, the processor is enabled to: adopt a data processing method based on a distributed computing model as described above.

[0067] This embodiment also provides a computer device, which includes: a processor; and a memory arranged to store computer executable instructions, and when the executable instructions are executed, the processor is enabled to: a data processing method based on a distributed computing model as described above.

[0068] The invention firstly supports various types of data processing based on a distributed computing model, pre-processes the collected IoT data, uses the KNN algorithm to remove noise data, uses singular value decomposition to reduce the dimension of the data, and reduces the execution time of data processing; then, a two-stage partitioning strategy is used for the processed data to ensure that the amount of data processed by each Reducer is as uniform as possible, thereby solving the problem of uneven load, dynamically adjusting the partition according to the actual load capacity, and optimizing resource utilization; through fine-grained partitioning and avoiding additional merging operations, the performance loss caused by data imbalance is reduced, and the overall processing efficiency is improved; then, by adopting a clustering analysis method, other server nodes with common time window intersections of the current distributed processing server nodes are found for task allocation, thereby improving data processing efficiency and resource utilization, and optimizing from two aspects of data partitioning and node selection; finally, a cuckoo algorithm is adopted in the Map and Reduce stages, and the convergence speed of the algorithm is improved as the number of iterative searches increases by attenuating the weight of the probability of finding a nest, thereby solving the problem that the clustering result is subject to the local optimum, thereby improving the accuracy and reliability of clustering.

[0069] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technical personnel in this field can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any brief modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A data processing method based on a distributed computing model, characterized in that: The following steps are involved: Data collection and preprocessing: Collect various forms of data from the Internet of Things and preprocess the collected data, including data cleaning and conversion, removing noise data and reducing the dimension of data; Data division and partitioning: Use hash algorithms to preliminarily partition data, calculate the load of each Reducer, identify and adjust abnormal partitions, refine abnormal partitions through fine-grained partitioning, and dynamically adjust partitioning strategies; Dynamic task allocation: According to the time window of the server node, identify the nodes with the same time window and generate an access policy set, and select the optimal access set for task allocation; Map and Reduce phase: Select core objects, expand clusters using density-based clustering algorithms, merge clustering results of all Map tasks, and use an iterative noise point processing strategy; Clustering result evaluation: Use evaluation indicators to evaluate clustering accuracy, analyze job execution time and load deviation, and verify the effectiveness of partitioning strategy and dynamic task allocation; Visualization: Select appropriate visualization tools to display clustering results, design chart types based on clustering results, and display clustering results in a graphical way.

2. The data processing method based on a distributed computing model according to claim 1, characterized in that: The data collection and preprocessing includes the following steps: Data cleaning and conversion: Identify missing values ​​in the data and fill them in by taking the mean; identify and delete duplicate records, use statistical methods to identify and process outliers, standardize numerical data, and convert categorical variables to numerical data; Clean up noisy data: Select the best K value through cross-validation. For each data point in the data set, calculate its distance to other data points, and for each data point, identify its K nearest neighbors. Check the distribution of each category label in the K neighbors or calculate the average of the values ​​of the K neighbors. If the similarity between a data point and its neighbors is significantly lower than that of most data points, or the frequency of the category label of the point in the neighbors is significantly lower, the data point is judged as noise and removed; Reduce the dimensionality of the data: Represent the original data set as a matrix, perform singular value decomposition on the data matrix, select the first K principal components based on the size of the singular values, and use the selected first K principal components to reconstruct the data set.

3. The data processing method based on a distributed computing model according to claim 1, characterized in that: The data division and partitioning includes the following steps: The first stage of hash partitioning: Use the traditional hash algorithm to process the input data and map the data to R partitions according to the key value. Each partition corresponds to a Reducer, and the data is distributed to different partitions. Set up data counters on each Mapper node to track the size of the data in each partition and calculate the data load received by each Reducer L r : ； In the formula, x ij Indicates i Mapper j Whether the partition is pulled, p ij Indicates the size of the partition; M Indicates the number of Mappers, and P indicates the number of partition types contained in each Mapper; Set a threshold. r The amount of data for each Reduce Lr If the value exceeds 90% of the threshold, it is considered an abnormal partition. Otherwise, it is pulled normally by the Reducer for calculation and processing.

4. The data processing method based on a distributed computing model according to claim 3, characterized in that: The data division and partitioning also includes the steps of: Second stage hash partitioning: According to the abnormal partitions identified in the first stage, each abnormal partition is subdivided into b groups; the current load capacity of each Reducer is calculated, and the Reducer with the best current load capacity is selected for each subdivided group using a greedy algorithm; Calculate the offset between the current optimal Reducer and the original partition number of the group , send the offset value to the Partitioner function, dynamically modify the hash partition function, and the modified Partioner function is: ； In the formula, is the modified Partitioner function, is the original Partitioner function; Merge partition results: Merge the partition results of the first stage with the partition results of the second stage to form the final partition file. The partition file contains the final partition information corresponding to each key value. According to the final partition file, the overall data is directly assigned to each Reducer for processing.

5. The data processing method based on a distributed computing model according to claim 1, characterized in that: The dynamic task allocation comprises the following steps: Establishing the original set: Establishing an original set that includes all distributed processing server nodes; Benchmark node selection: Take the first server node in the set as the benchmark, compare it with other nodes, and select all other server nodes that have a common time window with the benchmark node to form a new set; Intersection calculation: among the selected server nodes, continue to calculate the intersection of the time windows and gradually expand the set until all server nodes related to the benchmark node are found; Set update: After completing the processing of a benchmark node, remove the node from the original set, continue to select the next node as the new benchmark, and repeat the above process until all nodes are processed to generate the final multiple access policy sets.

6. The data processing method based on a distributed computing model according to claim 5, characterized in that: The dynamic task allocation further comprises the steps of: Select an access policy set: From the generated access policy set, select the access policy that passes through the most server nodes and has the shortest communication line; Delete invalid policies: Delete all access policies that include the server node selected in the previous step from the access policy set; Check coverage: Check whether the server nodes in all current access policies cover all server nodes; If yes, then end this phase and output the result; if no, then reselect the access strategy; Iterative selection: Repeat the above process until all server nodes have been visited; Output results: The final output is the selected optimal access set, including the order information of the accessed server nodes, as well as the corresponding communication path and resource consumption.

7. The data processing method based on a distributed computing model according to claim 1, characterized in that: The Map and Reduce phases include the following steps: Core object selection: Each Map task reads data points from a partition and stores them in the form of feature vectors. Each data point is processed using the improved cuckoo algorithm, which generates random numbers to explore the data space and identify the nest location of each data point. Calculate the objective function value for each data point, which includes: Density: Calculates the number of data points in the neighborhood around a data point; Distance: Calculate the Euclidean distance between a data point and other data points; Selection criteria for core objects: According to the calculated objective function value, the data points with the highest density or the best distance characteristics are selected as core objects as the basis for expanding the clustering cluster; Cluster expansion: For each core object, use the density reachability principle of the DBSCAN algorithm to check the data points in its neighborhood, calculate the number of data points in the neighborhood, and use a preset threshold to judge; If the number of data points in the neighborhood exceeds the set threshold, these points are classified into the cluster to which the core object belongs; for each newly added point, the neighborhood check process is repeated and the cluster is continued to expand until no new points are added; Clustering results merge: In the Reduce task, the clustering results of all Map tasks are collected, and each Map task outputs its own cluster and its member data points; check whether the clusters from different Map tasks overlap, and if there are common data points, merge them into one cluster; Use distance metrics to determine the attribution of data points, ensure that data points can be correctly classified into corresponding clusters, and finally generate a global clustering structure that contains the clustering results of all Map tasks; Noise point processing: In the merged clustering results, noise points are identified and processed using an iterative strategy. The noise points are classified into other clusters by re-evaluating their neighborhoods. If some noise points find new neighborhood data points in subsequent iterations, they are reclassified as cluster members. Regularly check and remove noise points in the clustering results, and dynamically adjust the noise point processing strategy by setting thresholds and iteration times.

8. The data processing method based on a distributed computing model according to claim 1, characterized in that: The clustering result evaluation comprises the following steps: Clustering accuracy evaluation: The quality of clustering results is evaluated by calculating the silhouette coefficient: For each data point , calculate the average distance between it and other points in the same cluster d ( ), calculate the average distance between the point and the points in the nearest neighbor cluster k ( ), silhouette coefficient s ( ) is calculated as: ； Execution time evaluation: record the start and end time of the Map phase and the Reduce phase, calculate the execution time of each phase separately to get the final total execution time; Load deviation analysis: Calculate the amount of data processed by each Map task and Reduce task, calculate the average value of the task load, and then calculate the standard deviation of the load deviation LD : ； In the formula, D ( h ) is the h The load of the task, n is the total number of tasks, is the average load of all tasks; Verify the effectiveness of partitioning strategy and dynamic task allocation: Check the amount of data read by each Map task and the execution time of each task to determine the uniformity of the data partitioning strategy; After checking the task allocation, the effectiveness of dynamic task allocation is judged by observing the differences in execution time of different tasks.

9. A computer-readable storage medium having computer-executable instructions stored thereon, wherein: When the computer executable instructions are executed by a processor, the processor is caused to: adopt the method according to any one of claims 1-8.

10. A computer device, wherein: include: A processor and a memory arranged to store computer executable instructions, which when executed cause the processor to: perform the method according to any one of claims 1 to 8.

Images