Processing method and processing device for Internet of Things data, and electronic equipment
By preprocessing the IoT data and iterative update of the MiniBatch K-Means algorithm, combined with the processing window recording error squared sum judgment convergence, the problem of iterative convergence judgment accuracy in IoT data clustering analysis is solved, and the analysis accuracy is improved .
Patent Information
- Application Number
- CN202411023147.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-11-08
AI Technical Summary
The accuracy of the iterative convergence judgment in the clustering analysis of IoT data in the existing technology is insufficient, resulting in the clustering results deviating from the true situation and affecting the accuracy of the analysis.
By preprocessing the IoT data, dividing the data set, and using the MiniBatch K-Means algorithm to randomly select sub-data sets and iteratively update the cluster center. The processing window uses the processing window to record the error squared sum of multiple iterations to judge iterative convergence, ensuring clustering Accuracy of the results.
The accuracy of judging the iterative update cluster center process is improved, making the clustering results closer to the real situation, thereby improving the accuracy of IoT data analysis.
Smart Images

Figure CN118916727A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a method and device for processing Internet of Things data, and an electronic device. Background Art
[0002] With the application of IoT technology in various industries (for example, in smart home systems), IoT data has shown explosive growth. These massive data contain multi-faceted information, which is very important for extracting valuable information and making intelligent decisions.
[0003] In the related art, clustering algorithms (e.g., K-means algorithm) are usually used to perform clustering analysis on the collected IoT data, and the IoT data is divided into different groups or categories, so that users can have a deeper understanding of the characteristics of the data. In addition, in the process of clustering analysis of IoT data, it is necessary to use an iterative algorithm to repeatedly iteratively update the cluster center to ensure the accuracy of clustering fitting of IoT data.
[0004] In the process of implementing the embodiments of the present disclosure, it is found that there are at least the following problems in the related art:
[0005] The solutions of related technologies are insufficient in the accuracy of convergence judgment during the iteration process, which may cause the clustering results to deviate from the actual situation and affect the accuracy of IoT data analysis.
[0006] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention
[0007] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.
[0008] The disclosed embodiments provide a method and device for processing IoT data, and an electronic device. When clustering IoT data, the accuracy of judging whether the iterative update of cluster centers is convergent can be improved, making the clustering results closer to the actual situation, thereby improving the accuracy of IoT data analysis.
[0009] In some embodiments, a method for processing IoT data includes: preprocessing the collected IoT data to obtain a standard data set, and dividing the standard data set into multiple sub-data sets; using a first setting algorithm to perform clustering fitting on the standard data set to obtain an initial cluster center; using a second setting algorithm to randomly select a sub-data set to iteratively update the initial cluster center, and using a processing window to calculate and record the sum of square errors of each iteration; confirming the convergence of the iteration based on the sum of square errors of multiple iterations recorded in the processing window, and terminating the iteration when the iteration convergence is confirmed, using the initial cluster center after the current iteration update as the target cluster center, and using the target cluster center to cluster the standard data set to obtain a clustering result.
[0010] Optionally, sub-datasets are randomly selected to iteratively update the initial cluster centers, including: in each iteration, a set number of sub-datasets are randomly selected to parallely calculate the intermediate cluster centers corresponding to each sub-dataset; and the initial cluster centers obtained in the previous iteration are updated according to the multiple intermediate cluster centers calculated in the current iteration.
[0011] Optionally, during each iteration, a set number of sub-datasets are randomly selected and the intermediate cluster centers corresponding to each sub-dataset are calculated in parallel, including: during each iteration, a KD-Tree is constructed based on the initial cluster centers obtained in the previous iteration; based on the KD-Tree, the data points in each sub-dataset are assigned to the initial cluster centers closest to it; for each initial cluster center, the mean of all data points assigned to it is calculated, and the intermediate cluster centers corresponding to each sub-dataset are calculated based on the mean.
[0012] Optionally, based on multiple intermediate cluster centers calculated in the current iteration, the initial cluster centers obtained in the previous iteration are updated, including: comparing the multiple intermediate cluster centers with the initial cluster centers obtained in the previous iteration, merging cluster centers whose distances are less than or equal to the set distance, and ensuring that the number of merged cluster centers is the set number during the merging process, to obtain the initial cluster centers of the current iteration.
[0013] Optionally, the collected IoT data is preprocessed to obtain a standard data set, including: removing rows containing null values for set features in the IoT data; calculating the standard score of each feature in the IoT data, and removing rows in the IoT data where the absolute value of the standard score on any feature is greater than a score threshold; and using PowerTransformer to standardize the features in the IoT data to obtain a standard data set.
[0014] Optionally, the convergence of the iteration is confirmed based on the sum of squared errors of multiple iterations recorded in the processing window, including: when the sum of squared errors of the current iteration is less than a tolerance threshold, the iteration convergence is confirmed; or, when the difference between the sum of squared errors of two adjacent iterations in multiple iterations is less than a set difference, the iteration convergence is confirmed.
[0015] Optionally, after obtaining the clustering results, the processing method also includes: calculating the sum of squared errors for the clustering results, and / or calculating the silhouette coefficient for the clustering results, and / or calculating the clustering separation for the clustering results; evaluating the effect and quality of this clustering based on the sum of squared errors, and / or the silhouette coefficient, and / or the clustering separation of the clustering results.
[0016] Optionally, after obtaining the clustering results, the processing method further includes: performing dimensionality reduction processing on the clustering results and generating a clustering result graph; converting the cluster centers in the clustering result graph from a standardized scale to an original scale to verify the clustering results.
[0017] In some embodiments, a processing device for IoT data includes a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the above-mentioned processing method for IoT data through the computer program.
[0018] In some embodiments, an electronic device includes: a device body; and a processing device for IoT data as described above, installed in the device body.
[0019] The method, device, and electronic device for processing IoT data provided by the embodiments of the present disclosure can achieve the following technical effects:
[0020] In the embodiment of the present disclosure, the sum of squared errors of multiple iterations recorded in the processing window is used to determine whether the iterations have converged. Since the processing window can smooth the fluctuations of the sum of squared errors during the iteration process and can reflect the changing trend of the sum of squared errors of multiple iterations, this is more accurate than the related art that only uses a single sum of squared errors to determine the convergence of the iterations. Therefore, when performing cluster analysis on IoT data, the embodiment of the present disclosure can improve the accuracy of judging whether the process of iteratively updating the cluster center has converged, making the clustering results closer to the actual situation, thereby improving the accuracy of IoT data analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0023] Figure 1 It is a schematic diagram of a hardware environment applicable to a method for processing Internet of Things data provided by an embodiment of the present disclosure;
[0024] Figure 2 is a schematic diagram of an electronic device provided by an embodiment of the present disclosure;
[0025] Figure 3 is a schematic diagram of a method for processing Internet of Things data provided by an embodiment of the present disclosure;
[0026] Figure 4 is a schematic diagram of another method for processing Internet of Things data provided by an embodiment of the present disclosure;
[0027] Figure 5 is a schematic diagram of another method for processing Internet of Things data provided by an embodiment of the present disclosure;
[0028] Figure 6 is a schematic diagram of another method for processing Internet of Things data provided by an embodiment of the present disclosure;
[0029] Figure 7 is a schematic diagram of a device for processing Internet of Things data provided by an embodiment of the present disclosure;
[0030] Figure 8 It is a schematic diagram of another device for processing Internet of Things data provided by an embodiment of the present disclosure. DETAILED DESCRIPTION
[0031] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0032] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0033] According to one aspect of the embodiments of the present disclosure, a method for processing IoT data is provided. The processing method is widely used to process IoT data of systems such as smart homes, smart homes, smart home device ecosystems, and smart residential ecosystems.
[0034] Optionally, in the embodiment of the present disclosure, the above-mentioned method for processing IoT data can be applied to: Figure 1 In the hardware environment composed of the electronic device 100 and the server 200 of the Internet of Things system shown in FIG. Figure 1 As shown, the electronic device 100 is connected to the server 200 of the Internet of Things system through a network, so the electronic device 100 can obtain the Internet of Things data to be processed from the server 200 of the Internet of Things system through the network.
[0035] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth.
[0036] The electronic device 100 may include but is not limited to a PC (Personal Computer), a mobile phone, and a tablet computer.
[0037] The above-mentioned smart home (Smart Home), smart home, smart home equipment ecosystem, smart residential (Intelligence House) ecosystem and other systems include: smart air conditioners, smart range hoods, smart refrigerators, smart ovens, smart stoves, smart washing machines, smart water heaters, smart washing equipment, smart dishwashers, smart projection equipment, smart TVs, smart clothes drying racks, smart curtains, smart audio and video, smart sockets, smart speakers, smart speakers, smart fresh air equipment, smart kitchen and bathroom equipment, smart bathroom equipment, smart sweeping robots, smart window cleaning robots, smart mopping robots, smart air purification equipment, smart steamers, smart microwave ovens, smart kitchen treasures, smart purifiers, smart water dispensers, smart door locks and other home appliances.
[0038] like Figure 2 As shown, the electronic device 100 includes: a device body 110 and a processing device 700 ( 800 ) for IoT data. The processing device 700 ( 800 ) for IoT data is installed in the device body 110 .
[0039] Optionally, the processing device 800 for IoT data includes a processor, which can preprocess the collected IoT data to obtain a standard data set, and divide the standard data set into multiple sub-data sets. The first setting algorithm can be used to perform cluster fitting on the standard data set to obtain an initial cluster center. The second setting algorithm can be used to randomly select a sub-data set to iteratively update the initial cluster center, and the processing window can be used to calculate and record the sum of square errors of each iteration. The convergence of the iteration can be confirmed based on the sum of square errors of multiple iterations recorded in the processing window, and when the iteration convergence is confirmed, the iteration is terminated, the initial cluster center after the current iteration update is used as the target cluster center, and the target cluster center is used to cluster the standard data set to obtain a clustering result.
[0040] In combination with the above electronic device, the present disclosure provides a method for processing IoT data, such as Figure 3 As shown, the processing method includes:
[0041] S301, the processor preprocesses the collected IoT data to obtain a standard data set, and divides the standard data set into multiple sub-data sets.
[0042] Specifically, it is understandable that there may be a large amount of duplicate data, missing values, and outliers in the IoT data, which are not conducive to subsequent cluster analysis. And the IoT data is collected from multiple objects in the system where the IoT is located (for example, from multiple home appliances in a smart home system, and for example, from different dimensions in a car networking system). Therefore, the IoT data includes data of multiple different dimensions, which is also not conducive to subsequent cluster analysis. Therefore, after acquiring the IoT data, the processor needs to pre-process the collected IoT data to obtain a standard data set.
[0043] Specifically, since a small batch of data subsets is required to reduce the computing time later, after obtaining the standard data set, the processor needs to divide the standard data set into a plurality of sub-data sets.
[0044] S302: The processor performs cluster fitting on the standard data set using a first set algorithm to obtain initial cluster centers.
[0045] Optionally, the first setting algorithm is a K-means++ algorithm.
[0046] Specifically, by using the K-means++ algorithm to perform cluster fitting on the standard data set and determine the initial cluster centers, it is possible to better ensure that the initial cluster centers are relatively evenly distributed in the data space of the standard data set. This is conducive to improving the efficiency of subsequently using the second setting algorithm to randomly select sub-data sets to iteratively update the initial cluster centers.
[0047] S303, the processor adopts the second setting algorithm, randomly selects a sub-data set to iteratively update the initial cluster center, and uses a processing window to calculate and record the sum of squared errors of each iteration.
[0048] Optionally, the second setting algorithm is a MiniBatch K-Means algorithm.
[0049] Specifically, since the amount of IoT data is relatively large, if a traditional clustering algorithm is used to iteratively update the cluster center for the entire standard data set, it will take a long time to solve the target cluster center. Therefore, the disclosed embodiment adopts the MiniBatch K-Means algorithm to iteratively update the initial cluster center by randomly selecting a sub-data set. Since the MiniBatch K-Means algorithm only needs to process a small part of the data instead of all the data each iteration. Therefore, it can shorten the time spent on solving the cluster center and improve the efficiency of cluster analysis of IoT data.
[0050] Specifically, when the MiniBatch K-Means algorithm is used as the second setting algorithm, the size of each sub-dataset is determined by the batch_size in the MiniBatch K-Means algorithm. Specifically, the standard data set can be divided into multiple sub-datasets using a random sampling method according to the size of the sub-dataset set by batch_size. The data in the standard data set can also be sorted by time, and then the standard data set can be divided into multiple sub-datasets according to the size of the sub-dataset set by batch_size. The embodiment of the present disclosure does not specifically limit the way to divide the standard data set.
[0051] It is understandable that before using the MiniBatch K-Means algorithm to iteratively solve the cluster centers, it is necessary to set various parameters of the MiniBatch K-Means algorithm, such as the number of cluster centers, the size of the sub-datasets, and the maximum number of iterations. Specifically, after the various parameters of the MiniBatch K-Means algorithm are set, a clustering model will be created based on these parameters for executing subsequent iterations.
[0052] Exemplarily, the parameters of the MiniBatch K-Means algorithm may be set as shown in Table 1 below:
[0053] Table 1
[0054]
[0055] Optionally, the processing window is a sliding window, a rolling window or a global window. In some embodiments, the sliding window is set as the processing window.
[0056] Specifically, by using a processing window to calculate and record the sum of squared errors (SSE) of each iteration, the fluctuation of the sum of squared errors during the iteration process can be smoothed, and the changing trend of the sum of squared errors of multiple iterations can be reflected, which helps to improve the accuracy of judging whether the iteration has converged.
[0057] In addition, due to the influence of factors such as noise data in IoT data and errors in clustering models, the sum of squared errors of a single iteration may be abnormal (lower than the tolerance threshold). By recording the sum of squared errors of multiple iterations in the processing window, the influence of these abnormal values on the accuracy of judging convergence can be reduced.
[0058] S304, the processor confirms the convergence of the iteration according to the sum of squared errors of multiple iterations recorded in the processing window, and terminates the iteration when the iteration convergence is confirmed, takes the initial cluster center updated in the current iteration as the target cluster center, and uses the target cluster center to cluster the standard data set to obtain the clustering result.
[0059] Specifically, according to the error sum of squares of multiple iterations recorded in the processing window, the change trend of the error sum of squares of recent iterations can be determined. If the error sum of squares of recent iterations gradually decreases and tends to be stable, it indicates that the iteration has converged. Therefore, the processor can confirm the convergence of the iteration according to the error sum of squares of multiple iterations recorded in the processing window.
[0060] Specifically, if it is confirmed that the iteration has converged, it can be determined that the accuracy of the initial cluster center obtained in the current iteration is high enough to meet the set requirements. In this case, the initial cluster center updated in the current iteration can be used as the target cluster center, and the target cluster center can be used to cluster the standard data set to obtain the clustering result of the Internet of Things data.
[0061] It can be understood that if the number of iterations reaches the maximum number of iterations, even if the iterations have not converged, the iterations will be terminated, and the standard data set will be re-divided, and the above steps will be re-executed until the iterations converge.
[0062] In the disclosed embodiment, the collected IoT data will first be preprocessed to obtain a standard data set, and the standard data set will be divided into multiple sub-data. Then, the first setting algorithm will be used to perform cluster fitting on the standard data set to obtain the initial cluster center; and the second setting algorithm will be used to randomly select a sub-data set to iteratively update the initial cluster center, and the sum of square errors of each iteration will be calculated and recorded through the pre-built processing window at each iteration. Finally, the convergence of the iteration will be confirmed according to the sum of square errors of multiple iterations recorded in the processing window, and when the iteration converges, the iteration will be terminated, and the initial cluster center updated in the current iteration will be used as the target cluster center, and the target cluster center will be used to cluster the standard data set to obtain the clustering result. It can be seen that the disclosed embodiment uses the sum of square errors of multiple iterations recorded in the processing window to determine whether the iteration converges. Since the processing window can smooth the fluctuation of the sum of square errors during the iteration process and can reflect the changing trend of the sum of square errors of multiple iterations, this is more accurate than the related art that only uses a single sum of square errors to judge the convergence of the iteration. Therefore, when performing cluster analysis on IoT data, the disclosed embodiment can improve the accuracy of judging whether the process of iteratively updating cluster centers has converged, making the clustering results closer to the actual situation, thereby improving the accuracy of IoT data analysis.
[0063] The present disclosure provides another method for processing IoT data. Figure 4 As shown, the processing method includes:
[0064] S401, the processor preprocesses the collected IoT data to obtain a standard data set, and divides the standard data set into multiple sub-data sets.
[0065] S402: The processor performs cluster fitting on the standard data set using a first set algorithm to obtain initial cluster centers.
[0066] S403, the processor adopts the second set algorithm to randomly select a set number of sub-data sets in each iteration process and parallelly calculate the intermediate cluster center corresponding to each sub-data set.
[0067] S404, the processor updates the initial cluster center obtained in the previous iteration according to the multiple intermediate cluster centers calculated in the current iteration, and calculates and records the sum of square errors of each iteration using a processing window;
[0068] Specifically, by calculating the corresponding intermediate cluster centers for a set number of sub-datasets in parallel during each iteration, and updating the initial cluster centers obtained in the previous iteration based on multiple intermediate cluster centers, the process of updating the initial cluster centers can be accelerated and the convergence speed of the iteration can be improved.
[0069] S405, the processor confirms the convergence of the iteration according to the sum of squared errors of multiple iterations recorded in the processing window, and terminates the iteration when the iteration convergence is confirmed, takes the initial cluster center updated in the current iteration as the target cluster center, and uses the target cluster center to cluster the standard data set to obtain the clustering result.
[0070] In the disclosed embodiment, by calculating intermediate cluster centers for a set number of sub-data sets in parallel during each iteration and updating the initial cluster centers obtained in the previous iteration based on multiple intermediate cluster centers, the rate of updating the initial cluster centers is improved, the convergence speed of the iteration is improved, and the overall efficiency of clustering analysis of IoT data is improved.
[0071] In some embodiments, a set number of sub-datasets are randomly selected and the intermediate cluster centers corresponding to each sub-dataset are calculated in parallel, including: in each iteration, a KD-Tree is constructed based on the initial cluster centers obtained in the previous iteration; based on the KD-Tree, the data points in each sub-dataset are assigned to the initial cluster centers closest to it; for each initial cluster center, the mean of all data points assigned to it is calculated, and the intermediate cluster center corresponding to each sub-dataset is calculated based on the mean.
[0072] Specifically, KD-Tree is an effective data structure for organizing points in k-dimensional space, which can realize fast search of multi-dimensional spatial data by continuously selecting dimensions for segmentation.
[0073] Specifically, in each iteration, by constructing the initial cluster center obtained in the previous iteration into a KD-Tree, the current iteration can more quickly and accurately assign the data points in each sub-dataset to the initial cluster center closest to it. In this way, the efficiency of searching for data points for each initial cluster center can be improved.
[0074] Specifically, the mean of all data points assigned to each initial cluster center is the intermediate cluster center corresponding to each sub-dataset.
[0075] In the disclosed embodiment, in each iteration process, a KD-Tree is constructed using the initial cluster centers obtained in the previous iteration, and based on the KD-Tree, the data points in each sub-dataset are assigned to the initial cluster center closest to it, thereby improving the efficiency of searching for data points for each initial cluster center, improving the speed of calculating intermediate cluster centers, and thereby improving the rate of updating the initial cluster centers.
[0076] In some embodiments, the initial cluster centers obtained in the previous iteration are updated based on multiple intermediate cluster centers calculated in the current iteration, including: comparing the multiple intermediate cluster centers with the initial cluster centers obtained in the previous iteration, merging cluster centers whose distances are less than or equal to a set distance, and ensuring that the number of merged cluster centers is a set number during the merging process, to obtain the initial cluster centers of the current iteration.
[0077] Specifically, by comparing multiple intermediate cluster centers with the initial cluster centers obtained in the previous iteration, it is possible to determine which cluster centers are closer to the data points and are more suitable as cluster centers. Therefore, the processor needs to compare multiple intermediate cluster centers with the initial cluster centers obtained in the previous iteration.
[0078] Specifically, it is understood that when cluster analysis is performed on IoT data, the number of cluster centers is given. Therefore, in the process of merging cluster centers whose distance is less than or equal to the set distance, it is necessary to ensure that the number of cluster centers after merging is the set number.
[0079] In the disclosed embodiment, in each iteration, it is necessary to compare multiple intermediate cluster centers with the initial cluster centers obtained in the previous iteration to determine which cluster centers are closer to the data points and more suitable as cluster centers, so as to merge the cluster centers with closer distances, and ensure that the number of merged cluster centers is the set number during the merging process. In this way, the accuracy of the initial cluster centers obtained in each iteration is improved.
[0080] In some embodiments, the collected IoT data is preprocessed to obtain a standard data set, including: removing rows containing null values for set features in the IoT data; calculating the standard score for each feature in the IoT data, and removing rows in the IoT data where the absolute value of the standard score on any feature is greater than a score threshold; and using PowerTransformer to standardize the features in the IoT data to obtain a standard data set.
[0081] Specifically, the features of IoT data represent the columns in the IoT dataset, which are used to describe the information or attributes of each row in the IoT dataset.
[0082] Specifically, the pandas tool can be used to load IoT data, and the dropna method can be used to remove rows containing null values based on the set features in the IoT data.
[0083] Specifically, the standard score (Z score) of each feature is equal to the deviation of each value in the feature from the mean divided by the standard deviation. Specifically, if the absolute value of the standard score on any feature of a row in the IoT data is greater than the score threshold, it indicates that this row contains outliers or erroneous data. Therefore, it is necessary to remove rows in the IoT data where the absolute value of the standard score on any feature is greater than the score threshold.
[0084] Optionally, the score threshold is set to 3.
[0085] Specifically, PowerTransformer is a preprocessing tool in the scikit-learn library, which is mainly used to transform data into a shape closer to a normal distribution through power transformation (such as Box-Cox transformation or Yeo-Johnson transformation).
[0086] Specifically, by using PowerTransformer to standardize the features of IoT data after removing null values and outliers, data of different dimensions in IoT data can be transformed into the same dimension, that is, the IoT data is converted from their original scales to standardized scales. This facilitates the subsequent clustering analysis of IoT data.
[0087] In the embodiment of the present disclosure, in the process of converting IoT data into a standardized data set, null values and outliers in the IoT data are removed, and the IoT data without null values and outliers is standardized. In this way, the accuracy of subsequent cluster analysis of IoT data can be further improved.
[0088] In some embodiments, the convergence of the iteration is confirmed based on the sum of squared errors of multiple iterations recorded in the processing window, including: when the sum of squared errors of the current iteration is less than a tolerance threshold, the iteration convergence is confirmed; or, when the difference between the sum of squared errors of two adjacent iterations in multiple iterations is less than a set difference, the iteration convergence is confirmed.
[0089] Specifically, if the sum of squared errors of the current iteration is less than the tolerance threshold, it indicates that the distances between the multiple data points in the processing window and the cluster center all meet the convergence condition. Therefore, in this case, it can be confirmed that the iteration has converged.
[0090] Specifically, if the difference between the sum of square errors of two adjacent iterations in multiple iterations is less than the set difference, it indicates that the sum of square errors of recent iterations is gradually decreasing and tends to be stable. Therefore, in this case, it can be confirmed that the iteration has converged.
[0091] In the disclosed embodiment, whether the iteration has converged is confirmed based on the sum of squared errors of multiple iterations recorded within the processing window. This improves the accuracy of judging whether the iteration has converged compared to the solution in the related art that only uses a single sum of squared errors to judge the convergence of the iteration.
[0092] The present disclosure provides another method for processing IoT data. Figure 5 As shown, the processing method includes:
[0093] S501, the processor preprocesses the collected IoT data to obtain a standard data set, and divides the standard data set into multiple sub-data sets.
[0094] S502: The processor performs cluster fitting on the standard data set using a first set algorithm to obtain initial cluster centers.
[0095] S503, the processor adopts the second setting algorithm, randomly selects a sub-data set to iteratively update the initial cluster center, and uses a processing window to calculate and record the sum of squared errors of each iteration.
[0096] S504, the processor confirms the convergence of the iteration according to the sum of squared errors of multiple iterations recorded in the processing window, and terminates the iteration when the iteration converges. The initial cluster center updated in the current iteration is used as the target cluster center, and the target cluster center is used to cluster the standard data set to obtain the clustering result.
[0097] S505: The processor calculates the sum of squared errors for the clustering results, and / or calculates the silhouette coefficient for the clustering results, and / or calculates the cluster separation degree for the clustering results.
[0098] S506: The processor evaluates the effect and quality of the clustering according to the sum of squared errors of the clustering results, and / or the silhouette coefficient, and / or the cluster separation degree.
[0099] In the implementation of the present disclosure, the effect and quality of the clustering are evaluated by calculating the sum of squared errors and / or the silhouette coefficient and / or the cluster separation of the clustering results. In this way, when the effect and quality are poor, timely feedback can be given, and the various parameters of the second setting algorithm (MiniBatch K-Means) can be adjusted to re-cluster the IoT data.
[0100] The present disclosure provides another method for processing IoT data. Figure 6 As shown, the processing method includes:
[0101] S601, the processor preprocesses the collected IoT data to obtain a standard data set, and divides the standard data set into multiple sub-data sets.
[0102] S602: The processor performs cluster fitting on the standard data set using a first set algorithm to obtain initial cluster centers.
[0103] S603, the processor adopts the second setting algorithm, randomly selects a sub-data set to iteratively update the initial cluster center, and uses a processing window to calculate and record the sum of square errors of each iteration.
[0104] S604, the processor confirms the convergence of the iteration based on the sum of squared errors of multiple iterations recorded in the processing window, and terminates the iteration when the iteration convergence is confirmed, takes the initial cluster center updated in the current iteration as the target cluster center, and uses the target cluster center to cluster the standard data set to obtain the clustering result.
[0105] S605: The processor performs dimensionality reduction processing on the clustering result and generates a clustering result graph.
[0106] Specifically, PCA and t-SNE tools can be used to reduce the dimension of the clustering results, reducing the data of the clustering results to two dimensions. Matplotlib can be used to draw a clustering result graph, and the cluster centers need to be marked in the clustering result graph.
[0107] S606: The processor converts the cluster centers in the clustering result graph from the standardized scale to the original scale to verify the clustering result.
[0108] Specifically, since the IoT data is preprocessed into a standard data set during the cluster analysis process, before verifying the clustering results based on the clustering result graph, it is necessary to convert the cluster centers in the clustering result graph from the standardized scale to the original scale.
[0109] In the embodiment of the present disclosure, the clustering result is verified by drawing a clustering result graph.
[0110] Combination Figure 7 As shown, the embodiment of the present disclosure provides a processing device 700 for IoT data, including: a data processing module 701, a clustering module 702 and an iteration module 703. The data processing module 701 is configured to pre-process the collected IoT data to obtain a standard data set, and divide the standard data set into multiple sub-data sets. The clustering module 702 is configured to use a first setting algorithm to perform cluster fitting on the standard data set to obtain an initial cluster center. The iteration module 703 is configured to use a second setting algorithm to randomly select a sub-data set to iteratively update the initial cluster center, and use a processing window to calculate and record the sum of square errors of each iteration. The clustering module 702 is also configured to confirm the convergence of the iteration according to the sum of square errors of multiple iterations recorded in the processing window, and terminate the iteration when the iteration converges, and use the initial cluster center updated in the current iteration as the target cluster center, and use the target cluster center to cluster the standard data set to obtain a clustering result.
[0111] In the disclosed embodiment, the clustering module 702 determines whether the iteration has converged based on the sum of squared errors of multiple iterations recorded in the processing window. Since the processing window can smooth the fluctuations of the sum of squared errors during the iteration process and can reflect the changing trend of the sum of squared errors of multiple iterations, this is more accurate than the related art that only uses a single sum of squared errors to judge the convergence of the iteration. Therefore, when performing cluster analysis on IoT data, the disclosed embodiment can improve the accuracy of judging whether the process of iteratively updating the cluster center has converged, making the clustering results closer to the actual situation, thereby improving the accuracy of IoT data analysis.
[0112] In some embodiments, the iteration module 703 is specifically configured to randomly select a set number of sub-data sets and parallely calculate the intermediate cluster centers corresponding to each sub-data set during each iteration; and update the initial cluster centers obtained in the previous iteration based on the multiple intermediate cluster centers calculated in the current iteration.
[0113] In some embodiments, the iteration module 703 is specifically configured to construct a KD-Tree according to the initial cluster centers obtained in the previous iteration during each iteration; assign the data points in each sub-dataset to the initial cluster centers closest to it based on the KD-Tree; calculate the mean of all data points assigned to each initial cluster center, and calculate the intermediate cluster center corresponding to each sub-dataset based on the mean.
[0114] In some embodiments, the iteration module 703 is specifically configured to compare multiple intermediate cluster centers with the initial cluster centers obtained in the previous iteration, merge cluster centers whose distances are less than or equal to the set distance, and ensure that the number of merged cluster centers is the set number during the merging process to obtain the initial cluster centers of the current iteration.
[0115] In some embodiments, the data processing module 701 is specifically configured to remove rows containing null values for set features in the IoT data; calculate the standard score for each feature in the IoT data, and remove rows in the IoT data where the absolute value of the standard score on any feature is greater than a score threshold; and use PowerTransformer to standardize the features in the IoT data to obtain a standard data set.
[0116] In some embodiments, the clustering module 702 is specifically configured to confirm iterative convergence when the sum of squared errors of the current iteration is less than a tolerance threshold; or to confirm iterative convergence when the difference between the sum of squared errors of two adjacent iterations in multiple iterations is less than a set difference.
[0117] In some embodiments, the data processing module 701 is also configured to calculate the sum of squared errors for the clustering results, and / or calculate the silhouette coefficient for the clustering results, and calculate the cluster separation for the clustering results; and evaluate the effect and quality of this clustering based on the sum of squared errors, and / or the silhouette coefficient, and / or the cluster separation of the clustering results.
[0118] In some embodiments, the data processing module 701 is further configured to perform dimensionality reduction processing on the clustering results and generate a clustering result graph; and convert the cluster centers in the clustering result graph from a standardized scale to an original scale to verify the clustering results.
[0119] Combination Figure 8As shown, an embodiment of the present disclosure provides a processing device 800 for Internet of Things data, and the processing device 800 for Internet of Things data includes: a processor (processor) 801 and a memory (memory) 802. Optionally, the device may also include a communication interface (Communication Interface) 803 and a bus 804. Among them, the processor 801, the communication interface 803, and the memory 802 can communicate with each other through the bus 804. The communication interface 803 can be used for information transmission. The processor 801 can call the logic instructions in the memory 802 to execute the processing method for Internet of Things data of the above embodiment.
[0120] In addition, the logic instructions in the memory 802 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.
[0121] The memory 802 is a computer-readable storage medium that can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present disclosure. The processor 801 executes functional applications and data processing by running the program instructions / modules stored in the memory 802, that is, implements the processing method for IoT data in the above-mentioned embodiments.
[0122] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory 802 may include a high-speed random access memory and may also include a non-volatile memory.
[0123] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned processing method for Internet of Things data.
[0124] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A method for processing Internet of Things data, characterized in that: include: Preprocess the collected IoT data to obtain a standard data set, and divide the standard data set into multiple sub-data sets; The first setting algorithm is used to perform cluster fitting on the standard data set to obtain the initial cluster center; Using the second setting algorithm, randomly selecting a sub-data set to iteratively update the initial cluster center, and using a processing window to calculate and record the sum of squared errors of each iteration; The convergence of the iteration is confirmed according to the sum of square errors of multiple iterations recorded in the processing window. When the iteration converges, the iteration is terminated, the initial cluster center updated in the current iteration is used as the target cluster center, and the target cluster center is used to cluster the standard data set to obtain the clustering result.
2. The method for processing Internet of Things data according to claim 1, characterized in that: Randomly select sub-datasets to iteratively update the initial cluster center, including: In each iteration, a set number of sub-datasets are randomly selected and the intermediate cluster centers corresponding to each sub-dataset are calculated in parallel; According to the multiple intermediate cluster centers calculated in the current iteration, the initial cluster centers obtained in the previous iteration are updated.
3. The method for processing Internet of Things data according to claim 2, characterized in that: In each iteration, a set number of sub-datasets are randomly selected and the intermediate cluster centers corresponding to each sub-dataset are calculated in parallel, including: In each iteration, a KD-Tree is constructed based on the initial clustering center obtained in the previous iteration; based on the KD-Tree, the data points in each sub-dataset are assigned to the initial clustering center closest to it; For each initial cluster center, the mean of all data points assigned to it is calculated, and the intermediate cluster center corresponding to each sub-dataset is calculated based on the mean.
4. The method for processing Internet of Things data according to claim 2, characterized in that: According to the multiple intermediate cluster centers calculated in the current iteration, the initial cluster centers obtained in the previous iteration are updated, including: Compare multiple intermediate cluster centers with the initial cluster centers obtained in the previous iteration, merge the cluster centers whose distance is less than or equal to the set distance, and ensure that the number of merged cluster centers is the set number during the merging process to obtain the initial cluster centers of the current iteration.
5. The method for processing Internet of Things data according to any one of claims 1 to 4, characterized in that: Preprocess the collected IoT data to obtain a standard data set, including: Remove rows with null values for set features in IoT data; Calculate the standard score for each feature in the IoT data, and remove rows in the IoT data where the absolute value of the standard score on any feature is greater than the score threshold; PowerTransformer is used to standardize the features in IoT data to obtain a standard data set.
6. The method for processing Internet of Things data according to any one of claims 1 to 4, characterized in that: Confirm the convergence of the iteration based on the sum of squared errors of multiple iterations recorded in the processing window, including: When the sum of squared errors of the current iteration is less than the tolerance threshold, the iteration is confirmed to have converged; or, When the difference between the sum of square errors of two adjacent iterations in multiple iterations is less than the set difference, the iteration converges.
7. The method for processing Internet of Things data according to any one of claims 1 to 4, characterized in that: After obtaining the clustering results, the processing method also includes: Calculating the sum of squared errors for the clustering results, and / or calculating the silhouette coefficient for the clustering results, and / or calculating the cluster separation degree for the clustering results; The effect and quality of this clustering are evaluated based on the sum of squared errors and / or silhouette coefficient and / or cluster separation of the clustering results.
8. The method for processing Internet of Things data according to any one of claims 1 to 4, characterized in that: After obtaining the clustering results, the processing method also includes: Perform dimensionality reduction on the clustering results and generate a clustering result graph; The cluster centers in the clustering result graph are converted from the standardized scale to the original scale to verify the clustering results.
9. A device for processing Internet of Things data, comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to execute the method for processing Internet of Things data according to any one of claims 1 to 8 through the computer program.
10. An electronic device, characterized in that: include: Equipment body; The processing device for Internet of Things data as described in claim 9 is installed on the device body.
Citation Information
Cited By
Data processing method and system based on enterprise management and storage medium
CN119886889A
