An internet of things data monitoring method and system

Abstract

The application relates to an Internet of Things data monitoring method and system, which uses the best cluster number to divide the data of Internet of Things equipment into different clusters through a K-Means clustering algorithm, calculates the average value and standard deviation of each electrical variable in each cluster, and determines the normal range and abnormal threshold of each electrical variable in each cluster; the method can comprehensively consider the connection between the electrical variables to determine the normal range and abnormal threshold of the electrical variables in each cluster, thereby better realizing the monitoring and analysis of the state and performance of the equipment, and realizing the optimization and fault prevention of the equipment; the application also uses a silhouette coefficient as an evaluation index of the clustering effect to determine the best cluster number, avoids the interference of artificial experience and rules, and improves the intelligence and self-adaptive capability of the clustering; the application uses a visual tool to display the data and clustering results of the equipment, so that the user can intuitively understand the condition of the equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of Internet of Things, in particular, to an Internet of Things data monitoring method and system. BACKGROUND

[0002] Internet of Things (IoT) refers to connecting various smart devices through the Internet to realize the exchange and communication of information. Internet of Things devices can collect and transmit various data such as temperature, humidity, pressure, voltage, current, and power, which reflect the state and performance of the device, and are of great significance for the optimization and fault prevention of the device. However, the data of Internet of Things devices is large, complex, and variable, which is difficult to effectively monitor and analyze. Traditional data monitoring methods often rely on human experience and rules, lack of intelligence and adaptive ability, and cannot meet the needs of data monitoring of Internet of Things devices. SUMMARY

[0003] The present application relates to an Internet of Things data monitoring method and system, which includes a data acquisition module, a data preprocessing module, a data clustering module, a data monitoring module, a data analysis module, and a data display module. By using the K-Means clustering algorithm, the data of Internet of Things devices is divided into different clusters using the best cluster number, and the average value and standard deviation of each electrical variable in each cluster are calculated to determine the normal range and abnormal threshold of each electrical variable in each cluster. This method can comprehensively consider the relationship between each electrical variable, thereby better realizing the monitoring and analysis of the state and performance of the device, and realizing the optimization and fault prevention of the device. The silhouette coefficient is used as an evaluation index of clustering effect to determine the best cluster number, avoiding the interference of human experience and rules, and improving the intelligence and adaptive ability of clustering. Visualization tools are used to display the data and clustering results of the device, so that users can intuitively understand the situation of the device.

[0004] In order to achieve the above-mentioned purpose, the technical scheme of the present application is as follows:

[0005] An Internet of Things data monitoring method, characterized in that it comprises the following steps:

[0006] (1) Data acquisition: collecting data from Internet of Things devices to form a data set, the data set including: device identification, electrical variables, state information, the electrical variables including temperature, humidity, pressure, voltage, current, and power; the data set being composed of multiple data groups;

[0007] (2) Data preprocessing: preprocessing the data set, the preprocessing including: screening out data groups with normal state information, removing missing values and outliers, standardizing numerical variables, and encoding categorical variables.

[0008] (3) Data clustering: using K-Means clustering algorithm, selecting different cluster numbers, clustering the preprocessed data set, evaluating the clustering effect and determining the best cluster number according to the silhouette coefficient; using the best cluster number, clustering the preprocessed data;

[0009] (4) Data monitoring: for the clustering result, calculating the mean and standard deviation of each electrical variable in each cluster to determine the normal range and abnormal threshold of each electrical variable in each cluster; for new device data, judging which cluster it belongs to, then comparing the value of each electrical variable in the new device data with the normal range and abnormal threshold of each electrical variable in the belonging cluster to determine whether the device is abnormal.

[0010] Further, the method can further include the following steps:

[0011] (5) Data analysis: analyzing the data and clustering results of the device to find the patterns, advantages and disadvantages of the state and performance of the device, and proposing optimization suggestions for the device;

[0012] (6) Data display: using visualization tools to display the data and clustering results of the device, so that users can intuitively understand the situation of the device.

[0013] An Internet of Things data monitoring system, characterized in that it comprises the following modules:

[0014] Data acquisition module: acquiring data from Internet of Things devices to form a data set, the data set including: device identification, electrical variables, state information, the electrical variables including temperature, humidity, pressure, voltage, current, power; the data set being composed of multiple data groups;

[0015] Data preprocessing module: preprocessing the data set, the preprocessing including: screening out data groups with normal state information, removing missing values and outliers, standardizing numerical variables, and encoding categorical variables;

[0016] Data clustering module: using K-Means clustering algorithm, selecting different cluster numbers, clustering the preprocessed data set, evaluating the clustering effect and determining the best cluster number according to the silhouette coefficient; using the best cluster number, clustering the preprocessed data;

[0017] Data monitoring module: for the clustering results, calculate the mean and standard deviation of each electrical variable in each cluster to determine the normal range and abnormal threshold of each electrical variable in each cluster; for new device data, determine which cluster it belongs to, then compare the value of each electrical variable in the new device data with the normal range and abnormal threshold of each electrical variable in the belonging cluster to determine whether the device is abnormal.

[0018] Further, the system can further include the following modules:

[0019] Data analysis module: analyze the data of the device and the clustering results to find the patterns, advantages and disadvantages of the state and performance of the device, and propose optimization suggestions for the device;

[0020] Data display module: use visualization tools to display the data of the device and the clustering results, so that users can intuitively understand the situation of the device.

[0021] Further, the profile coefficient is:

[0022]

[0023] where x is a data point, a(x) is the average distance of x from other data points in its own category, b(x) is the average distance of x from data points in the nearest other category; s(x) is the profile coefficient of x, with a value range of [-1, 1], the closer to 1, the better the clustering effect.

[0024] Further, the determination of which cluster it belongs to is: calculate the distance of the new device data to each cluster center, select the category to which the nearest cluster center belongs as the cluster of the new device data.

[0025] Further, the normal range of each electrical variable in each cluster is: [μ-3σ, u+3σ]; the abnormal threshold of each electrical variable in each cluster is: [μ-4σ, u+4σ]; where μ is the average value of a certain electrical variable of the device belonging to the cluster, σ is the standard deviation of a certain electrical variable of the device belonging to the cluster.

[0026] Further, the encoding formula used by the encoding categorical variable is:

[0027]

[0028] where x is the original categorical variable, k is the number of categories, c i is the i-th category, I(x=c i ) is an indicator function, when x=c i1 if x is 0, otherwise 0, x' is the encoded numerical variable, using binary encoding, which can effectively distinguish different categories.

[0029] The beneficial effects of the present application are as follows:

[0030] (1) The present application uses the K-Means clustering algorithm to divide the data of Internet of Things devices into different clusters using the optimal number of clusters, calculates the mean and standard deviation of each electrical variable in each cluster to determine the normal range and abnormal threshold of each electrical variable in each cluster, judges which cluster the new device data belongs to, and then compares the value of each electrical variable in the new device data with the normal range and abnormal threshold of each electrical variable in the cluster to determine whether the device is abnormal; this method can comprehensively consider the relationship between each electrical variable to determine the normal range and abnormal threshold of electrical variables in each cluster, thereby better realizing the monitoring and analysis of the state and performance of the device, and realizing the optimization and fault prevention of the device;

[0031] (2) The present application uses the silhouette coefficient as an evaluation index of clustering effect to determine the optimal number of clusters, avoiding the interference of artificial experience and rules, and improving the intelligence and self-adaptive ability of clustering;

[0032] (3) The present application uses a visualization tool to display the data and clustering results of the device, so that the user can intuitively understand the situation of the device, facilitating the decision-making and management of the user. BRIEF DESCRIPTION OF DRAWINGS

[0033] Other features, objects and advantages of the present application will become more apparent from the following detailed description of non-limiting embodiments with reference to the attached drawings:

[0034] Figure 1 is a workflow diagram of the present application;

[0035] Figure 2 is a system structure diagram of the present application. DETAILED DESCRIPTION

[0036] The present application will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application. In addition, it should be understood that those skilled in the art can make various modifications or changes to the present application after reading the content taught by the present application, and these equivalent forms also fall within the scope defined by the present application.

[0037] Suppose there is a dataset of an Internet of Things device, containing the following fields: device_id, temperature, humidity, pressure, voltage, current, power, status. Among them, device_id is the identification of the device, temperature is the temperature of the device, unit is Celsius, humidity is the humidity of the device, unit is percentage, pressure is the pressure of the device, unit is Pa, voltage is the voltage of the device, unit is V, current is the current of the device, unit is A, power is the power of the device, unit is W, status is the status of the device, there are three possible values: normal (normal), warning (warning), error (error). Part of the example of the dataset is shown in the following table, which includes 6 data groups, two data groups of status information are normal, two data groups of status information are warning, and two data groups of status information are error.

[0038] device_id temperature humidity pressure volTage current power status 1 25 50 100 220 10 2200 normal 2 30 60 110 230 11 2530 normal 3 35 70 120 240 12 2880 warning 4 40 80 130 250 13 3250 warning 5 45 90 140 260 14 3640 error 6 50 100 150 270 15 4050 error

[0039] Table 1

[0040] As Figures 1-2 , the system of the present application includes data acquisition module, data preprocessing module, data monitoring module, data analysis module, data display module, the above-mentioned functional modules can execute the functions of the corresponding steps in the following method class. The working process of the Internet of Things data monitoring method and system of the present application is as follows:

[0041] (1) Data acquisition: collect data from Internet of Things devices to form a dataset, the dataset includes: identification of the device, electrical variables, status information, the electrical variables include temperature, humidity, pressure, voltage, current, power; the dataset is composed of multiple data groups;

[0042] (2) Data preprocessing: The data set is preprocessed, and the preprocessing includes screening data groups with normal state information, removing missing values and outliers, standardizing numerical variables, and encoding categorical variables. In order to better ensure the clustering effect, the preprocessed data set includes at least 100 data groups. Specifically, the data preprocessing module performs the following operations: as shown in Table 1, the data groups corresponding to device numbers 1 and 2 with normal state information are screened out. -Remove missing values: Check if there are missing values in the data set. If there are, delete the data rows containing missing values; -Remove outliers: Check if there are outliers in the data set, such as temperature less than 0 or greater than 100, humidity less than 0 or greater than 100, pressure less than 0 or greater than 200, voltage less than 0 or greater than 300, current less than 0 or greater than 20, power less than 0 or greater than 5000, etc. If there are, delete the data rows containing outliers; -Standardize numerical variables: Standardize numerical variables such as temperature, humidity, pressure, voltage, current, power, etc. That is, subtract the average value and divide by the standard deviation, so that the mean of each variable is 0 and the standard deviation is 1, which is convenient for subsequent clustering algorithm processing; -Encode categorical variables: Encode categorical variables such as state, that is, map them to numerical values, such as mapping normal to 0, warning to 1, and error to 2, which is convenient for subsequent clustering algorithm processing.

[0043] The formula for encoding is:

[0044]

[0045] where x is the original categorical variable, k is the number of categories, c i is the i-th category, I(x=c i ) is an indicator function, which is 1 when x=c i , otherwise 0, and x' is the encoded numerical variable. Using binary encoding, different categories can be effectively distinguished.

[0046] (3) Data clustering: Use the K-Means clustering algorithm to cluster the preprocessed data set with different cluster numbers, and evaluate the clustering effect and determine the best cluster number according to the silhouette coefficient; use the best cluster number to cluster the preprocessed data.

[0047] Specifically, the data clustering module performs the following operations: - using a clustering algorithm: the data clustering module uses the K-Means algorithm to cluster the preprocessed data, the basic idea of the K-Means algorithm is to give a cluster number k, randomly select k data points as the initial cluster centers, then calculate the distance of each data point to each cluster center, assign each data point to the class to which the nearest cluster center belongs, then update the cluster center of each class to the average value of the data points in that class, repeat this process until the cluster centers no longer change or the maximum number of iterations is reached; the formula for calculating the distance of each data point to each cluster center is:

[0048]

[0049] where x is a data point, c is a cluster center, n is the number of variables, x i and c i are the values of x and c on the i-th variable, d(x, c) is the Euclidean distance from x to c.

[0050] The data clustering module selects an appropriate cluster number k, k determines the granularity and effect of clustering, if k is too small, the clustering result may be too rough, if k is too large, the clustering result may be too detailed. The data clustering module uses the silhouette coefficient as an evaluation index of clustering effect, the silhouette coefficient reflects the comprehensive of the internal similarity of the data point to its own class and the external similarity to other classes, the larger the silhouette coefficient, the better the clustering effect. The data clustering module tries different k values from 2 to 10, then calculates the silhouette coefficient of the clustering result corresponding to each k value, selects the k value with the largest silhouette coefficient as the best cluster number; the formula for calculating the silhouette coefficient is:

[0051]

[0052] where x is a data point, a(x) is the average distance of x to other data points in its own class, b(x) is the average distance of x to the nearest data point in other classes, s(x) is the silhouette coefficient of x, the value range is [-1, 1], the closer to 1 indicates the better clustering effect.

[0053] The data clustering module uses the best cluster number k to cluster the preprocessed data, gets the cluster label of each data point, adds the cluster label to the data set as a new field.

[0054] 4) Data monitoring: for the clustering results, calculate the mean and standard deviation of each electrical variable in each cluster to determine the normal range and abnormal threshold of each electrical variable in each cluster; for new device data, determine which cluster it belongs to, and then compare the value of each electrical variable in the new device data with the normal range and abnormal threshold of each electrical variable in the belonging cluster to determine whether the device is abnormal. The normal range and abnormal threshold of the same electrical variable (such as temperature) calculated in different clusters may be different, so it is necessary to determine which cluster the new device data belongs to first, and then determine the normal range and abnormal threshold of the electrical variable. When the temperature range in the belonging cluster is high (i.e. the device is overheating), the abnormal threshold of current, voltage and power in this cluster will generally be low, because the device cannot run in high power mode when it is overheating; when the temperature range in the belonging cluster is low, the abnormal threshold of current, voltage and power in this cluster will generally be high. By using this determination method, the relationship between various electrical variables can be considered comprehensively to determine the normal range and abnormal threshold of the electrical variables.

[0055] Assuming that the data of the device follows a normal distribution, using 3 times the standard deviation as the normal range and using 4 times the standard deviation as the abnormal threshold; the normal range of the electrical variable can be determined as: [μ-3σ, u+3σ]; the abnormal threshold of the electrical variable can be determined as: [μ-4σ, u+4σ]; where μ is the mean of a certain electrical variable of the cluster to which the device belongs, and σ is the standard deviation of a certain electrical variable of the cluster to which the device belongs.

[0056] - Determine the cluster of new device data: the data monitoring module determines which cluster the new device data belongs to, i.e. calculates the distance from the new device data to each cluster center, selects the cluster center belonging to the class with the closest distance, and determines the cluster of the new device data;

[0057] - Compare the new device data with the normal range and abnormal threshold: the data monitoring module compares the new device data with the normal range and abnormal threshold of the cluster to which it belongs, and if it exceeds the threshold, it is considered that the device is abnormal and needs to be alarmed or repaired. Specifically, the data monitoring module compares each electrical variable of the new device data with the normal range and abnormal threshold of the electrical variable of the cluster to which it belongs, if it exceeds the normal range but is within the abnormal threshold, it indicates that the device is warning and needs attention, and if it exceeds the abnormal threshold, it indicates that the device is error and needs to be repaired.

[0058] Further, the method further comprises the following steps:

[0059] (5) Data analysis: The data analysis module analyzes the data and clustering results of the equipment to find the patterns of the state and performance of the equipment, advantages and disadvantages, and proposes optimization suggestions for the equipment. Specifically, the data analysis module performs the following operations:

[0060] - Find the patterns of the state and performance of the equipment: The data analysis module analyzes the data and clustering results of the equipment to find the patterns of the state and performance of the equipment, i.e., each cluster represents a pattern of the state and performance of the equipment, and the characteristics of the equipment in each cluster can be described according to the mean and standard deviation of the data in each cluster, such as temperature, humidity, pressure, voltage, current, power, state, warning, error, etc.

[0061] - Find the advantages and disadvantages of the equipment: The data analysis module analyzes the data and clustering results of the equipment to find the advantages and disadvantages of the equipment, i.e., the data of each equipment can be compared with the data of the cluster it belongs to to find the advantages and disadvantages of the equipment, such as whether the temperature of the equipment is appropriate, whether the humidity is suitable, whether the pressure is stable, whether the voltage is normal, whether the current is reasonable, whether the power is efficient, whether the state is normal, warning, error, etc.

[0062] - Propose optimization suggestions for the equipment: The data analysis module analyzes the data and clustering results of the equipment to propose optimization suggestions for the equipment, i.e., according to the advantages and disadvantages of the equipment, give improvement schemes for the equipment, such as adjusting the temperature, humidity, pressure, voltage, current, power, etc. parameters of the equipment to achieve the best state and performance of the equipment.

[0063] (6) Data display: The data display module uses visualization tools such as line charts, bar charts, scatter plots, etc. to display the data and clustering results of the equipment, so that users can intuitively understand the situation of the equipment. Specifically, the data display module performs the following operations:

[0064] - Draw a line chart: The data display module uses a line chart to display the data of the equipment, which can show the trend of the data of the equipment over time, such as the temperature, humidity, pressure, voltage, current, power, etc. curve over time, which can help users find fluctuations and abnormal situations in the data of the equipment;

[0065] - Draw a bar chart: The data display module uses a bar chart to display the data of the equipment, which can show the distribution and comparison of the data of the equipment, such as the mean, standard deviation, maximum, minimum, etc. statistical quantities of the temperature, humidity, pressure, voltage, current, power, etc. of the equipment, which can help users find the characteristics and differences of the data of the equipment;

[0066] - Scatter plot: the data display module uses scatter plots to display the data of the device. Scatter plots can show the relationship and clustering of the data of the device, such as the scatter plots of the temperature and humidity, pressure and voltage, current and power, etc. of the device, which can help users discover the correlation and patterns of the data of the device.

[0067] The above description is merely that of a specific implementation of the present disclosure, which enables those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the generic principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An Internet of Things data monitoring method, characterized by, It comprises the following steps: (1) Data collection: collecting data from Internet of Things devices to form a data set, the data set comprising: the identification of the device, electrical variables, state information, the electrical variables comprising temperature, humidity, pressure, voltage, current, power; the data set being composed of multiple data groups; (2) Data preprocessing: preprocessing the data set, the preprocessing comprising: screening data groups with normal state information, removing missing values and outliers, standardizing numerical variables, and encoding categorical variables; (3) Data clustering: using the K-Means clustering algorithm, selecting different cluster numbers, clustering the preprocessed data set, evaluating the clustering effect and determining the optimal cluster number according to the silhouette coefficient; using the optimal cluster number to cluster the preprocessed data; (4) Data monitoring: for the clustering results, calculating the mean and standard deviation of each electrical variable in each cluster to determine the normal range and abnormal threshold of each electrical variable in each cluster; for new device data, determining which cluster it belongs to, then comparing the value of each electrical variable in the new device data with the normal range and abnormal threshold of each electrical variable in the belonging cluster to determine whether the device is abnormal; the normal range and abnormal threshold of the same electrical variable calculated in different clusters include different cases; when the temperature range of a cluster is lower than that of another cluster, the abnormal threshold of the current, voltage and power corresponding to the cluster with a lower temperature range will be set to be higher than the abnormal threshold of the corresponding parameters in the cluster with a higher temperature range; The encoding formula used for the encoding of the categorical variables is: where x is the original categorical variable, k is the number of categories, c i is the i-th category, I(x = c i ) is the indicator function, which is 1 when x = c i , otherwise 0, and x' is the encoded numerical variable using binary encoding, which can effectively distinguish different categories.

2. The IoT data monitoring method of claim 1, wherein, The silhouette coefficient is: Where x is a data point, a(x) is the average distance of x from other data points in its own cluster, b(x) is the average distance of x from data points in the nearest other cluster; s(x) is the silhouette coefficient of x, with a value range of [-1, 1], and the closer to 1, the better the clustering effect.

3. The IoT data monitoring method of claim 1, wherein, The determination of which cluster it belongs to is as follows: calculating the distance of the new device data to each cluster center, selecting the cluster center belonging to the class with the closest distance, as the cluster of the new device data.

4. An Internet of Things data monitoring system characterized by, It comprises the following modules: Data collection module: collecting data from Internet of Things devices to form a data set, the data set comprising: the identification of the device, electrical variables, state information, the electrical variables comprising temperature, humidity, pressure, voltage, current, power; the data set being composed of multiple data groups; Data preprocessing module: preprocessing the data set, the preprocessing comprising: screening data groups with normal state information, removing missing values and outliers, standardizing numerical variables, and encoding categorical variables; Data clustering module: using the K-Means clustering algorithm, selecting different cluster numbers, clustering the preprocessed data set, evaluating the clustering effect and determining the optimal cluster number according to the silhouette coefficient; using the optimal cluster number to cluster the preprocessed data; Data monitoring module: for the clustering results, calculate the mean and standard deviation of each electrical variable in each cluster to determine the normal range and abnormal threshold of each electrical variable in each cluster; for new device data, determine which cluster it belongs to, then compare the value of each electrical variable in the new device data with the normal range and abnormal threshold of each electrical variable in the belonging cluster to determine whether the device is abnormal; the normal range and abnormal threshold of the same electrical variable calculated in different clusters include different cases; when the temperature range of a cluster is lower than that of another cluster, the abnormal threshold of current, voltage and power corresponding to the parameters of the cluster with lower temperature range will be set to be higher than the abnormal threshold of the corresponding parameters in the cluster with higher temperature range; The encoding formula used by the encoding categorical variable is: where x is the original categorical variable, k is the number of categories, c i is the i-th category, I(x = c i ) is the indicator function, which is 1 when x = c i , otherwise 0, and x' is the encoded numerical variable using binary encoding, which can effectively distinguish different categories.

5. The IoT data monitoring system of claim 4, wherein, The profile coefficient is: Wherein, x is a data point, a(x) is the average distance of x and other data points in its belonging category, b(x) is the average distance of x and the data points in the nearest other category; s(x) is the profile coefficient of x, the value range is [-1, 1], the closer to 1 indicates the better clustering effect.

6. The IoT data monitoring system of claim 4, wherein, The determination of which cluster it belongs to is: calculate the distance of the new device data to each cluster center, select the category of the cluster center with the closest distance as the cluster of the new device data.

Images