A multi-dimensional time series data cube Internet of Things data mining method and system
The multi-dimensional time series data cube method improves IoT data mining efficiency and accuracy by cleaning, encoding, and constructing a data cube for faster data association and visualization.
Patent Information
- Application Number
- CN202411004245.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-10
- Estimated Expiration
- 2044-07-25
AI Technical Summary
Existing data mining algorithms for IoT devices face challenges due to large volumes of diverse data types, noise, and low accuracy and speed, leading to inefficient data processing and analysis.
A multi-dimensional time series data cube approach that includes data cleaning, reshaping, encoding, and storage in a pre-built database, followed by constructing a multi-dimensional time series plane, mining for positive and negative deviations, and visualizing the results.
Enhances data mining efficiency and accuracy by quickly processing and associating IoT data in multiple dimensions, enabling faster and deeper data mining and visualization.
Smart Images
Figure CN118535620B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of electronic digital data processing, and in particular to a multi-dimensional time series data cube Internet of Things data mining method and system. Background Art
[0002] With the advancement of sensor technology, wireless communication technology, cloud computing and big data processing technology, the Internet of Things has developed rapidly. IoT devices can collect and transmit large amounts of data, which is of great value in improving business processes, increasing efficiency and innovating service models.
[0003] Connected devices are everywhere, from smart homes to industrial automation, from smart cities to smart agriculture, and the amount of data generated by these devices is growing exponentially. These data contain huge potential value, which needs to be extracted and utilized through data mining technology. However, there are great challenges in data mining. The amount of data generated by IoT devices is huge, and the data types are diverse, including structured, semi-structured and unstructured data. Effective data preprocessing, cleaning and conversion technologies are needed to improve data quality and ensure the accuracy of analysis results. However, in traditional data mining algorithms, due to various limitations such as noise, data accuracy, and calculation speed, data mining efficiency and accuracy are low. Summary of the invention
[0004] In order to efficiently process data and improve data mining efficiency, the present invention provides a multi-dimensional time series data cube Internet of Things data mining method and system.
[0005] The present invention provides a multi-dimensional time series data cube Internet of Things data mining method, which adopts the following technical solutions:
[0006] A multi-dimensional time series data cube Internet of Things data mining method comprises the following steps:
[0007] Get IoT data;
[0008] Extract the timeline, location information, data type and data value from IoT data, reshape and encode IoT data according to preset encoding rules, and perform data cleaning;
[0009] The IoT data that has completed the shaping encoding is divided into blocks and stored in a pre-built database according to the timeline, location information, data type and data value;
[0010] Read the IoT data in the database, build a multi-dimensional time series plane based on the timeline through the constructed algorithm model, define different dimensions, and build a multi-dimensional plane so that each multi-dimensional time series plane contains multi-dimensional data distribution;
[0011] Construct a data cube in a multi-dimensional space to mine IoT data;
[0012] The mining data generated by the mining is judged. If it is judged to be positive deviation data, it is stored in the pre-built positive deviation database; if it is judged to be negative deviation data, it is stored in the pre-built negative deviation database;
[0013] Call the data in the positive deviation database and the negative deviation database to output visualization graphics.
[0014] In a specific implementation scheme, the rule for shaping and encoding IoT data is:
[0015] The 1st to 4th characters are the year, the 5th to 6th characters are the month, the 7th to 8th characters are the day, the 9th to 12th characters are the time, the 13th to 16th characters are the data type, the 17th to 20th characters are the location information, and the 21st to 24th characters are the data value.
[0016] In a specific embodiment, the method for constructing a multidimensional plane is:
[0017] Predefine the timeline dimension as T, the data type dimension as D, and the location information dimension as S;
[0018] Each data type generates a dimensional plane in the time coordinate, and each location information generates a dimensional plane in the time coordinate, forming a multi-dimensional space;
[0019] In the multi-dimensional space, according to the time coordinate, a plurality of dimensional planes containing data types and a dimensional plane containing location information are generated;
[0020] The IoT data in the database is retrieved in a time series manner and stored as point data in a multi-dimensional space. Each dimensional plane contains multiple point data.
[0021] In a specific implementation scheme, the point data includes a timeline, location information, a data type, a data value, and a feature value.
[0022] In a specific feasible implementation plan, the method for mining IoT data is:
[0023] Taking the time dimension as the baseline, traverse the data cube, calculate the support of the preset item set, compare the support with the preset minimum support threshold, and generate two item sets as candidate frequent item sets based on the item set whose support is higher than the minimum support threshold. Repeat until no candidate frequent item sets can be generated.
[0024] Generate association rules through candidate frequent item sets, calculate the confidence and lift of each association rule, and compare them with the preset minimum confidence and minimum lift, and retain the association rules with confidence greater than or equal to the minimum confidence and lift greater than or equal to the minimum lift.
[0025] In a specific implementation scheme, the method for determining the mining data generated by mining is:
[0026] According to the preset judgment conditions, based on the expected effect, preset thresholds, pre-constructed models and data change trends, the data validity is judged and the data is divided into positive deviation data and negative deviation data.
[0027] The present invention provides a multi-dimensional time series data cube Internet of Things data mining system, which adopts the following technical solutions:
[0028] A multi-dimensional time series data cube Internet of Things data mining system, comprising:
[0029] Data collection components collect corresponding IoT data in the environment and upload them;
[0030] A memory and a processor, the memory and the processor are in communication connection, the memory stores instructions that can be executed by the processor, and after the instructions are executed by the processor, the processor is enabled to execute the above-mentioned multi-dimensional time series data cube Internet of Things data mining method;
[0031] The display is connected to the processor for displaying output visualization graphics.
[0032] In a specific possible implementation scheme, when there are multiple data acquisition elements, the times of different data acquisition elements are kept consistent.
[0033] In a specific possible implementation scheme, when there are multiple data acquisition elements, different data acquisition elements correspond to unique position information.
[0034] In summary, the present invention includes at least one of the following beneficial technical effects:
[0035] 1. Through the multi-dimensional time series data cube IoT data mining algorithm, under the same data volume, IoT data can be processed more quickly and efficiently, improving data mining efficiency.
[0036] 2. By constructing a multi-dimensional time series plane, the IoT data in the data set is converted into point data in the multi-dimensional time series plane. By constructing a data cube to build the association of point data in multiple multi-dimensional time series planes, the IoT data can be associated in different dimensions, thereby quickly and efficiently establishing IoT data associations and deeply mining IoT data. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1It is a flow chart of the multi-dimensional time series data cube IoT data mining method.
[0038] Figure 2 It is a flow chart of constructing a data cube from a multi-dimensional time series plane. DETAILED DESCRIPTION
[0039] The following is combined with Figure 1 The present invention is described in further detail.
[0040] Reference Figure 1 ,The multi-dimensional time series data cube IoT data mining method includes the following steps:
[0041] S100, obtain IoT data.
[0042] The time of the IoT platform, collectors, and sensors are consistent. All collectors and sensors have unique location information. For example, when the time of sensor A is 11:23, the time of the IoT platform and other collectors and sensors is also 11:23. If the location information of sensor A is A1A1, then the location information of other collectors or sensors cannot be A1A1, and the location information of sensor A is only A1A1, which will not be the same as the location information of other collectors or sensors.
[0043] S200, IoT data shaping encoding and data cleaning.
[0044] The rules for shaping IoT data are as follows: 1-4 characters are for the year, 5-6 characters are for the month, 7-8 characters are for the day, 9-12 characters are for the time, 13-16 characters are for the data type, 17-20 characters are for the location information, and 21-24 characters are for the data value. For example, if the code for an IoT data is: 20230101112300003A1A128c, it means that it was obtained at 11:23 on January 1, 2023, the data type is 0000, the location information is A1A1, and the data value is 128c.
[0045] Extract the timeline, location information, data type and data value from the IoT data, and perform shaping encoding on the IoT data according to the encoding rules. While encoding the IoT data, clean the IoT data to remove duplicate data, erroneous data and incomplete data.
[0046] S300, IoT data block storage.
[0047] Build a Hadoop database and store the IoT data that has completed data cleaning in the Hadoop database in blocks according to time, location information, data type, and data value.
[0048] S400, constructing a multidimensional time series plane through a multidimensional time series space construction algorithm. According to the multidimensional time series algorithm, a multidimensional space is constructed, and the IoT data in the Hadoop database forms point data in the multidimensional space.
[0049] Based on the timeline, multi-dimensional time series planes such as data type dimension, location information dimension, and data value dimension are constructed in sequence. That is, a two-dimensional coordinate system is constructed with the timeline and data type, and a two-dimensional coordinate system is constructed with the timeline and location information.
[0050] The IoT data in the Hadoop database is retrieved in chronological order, and the IoT data is stored as point data in a multidimensional time series plane, that is, the IoT data is converted into points in a multidimensional time series plane. Each multidimensional time series plane contains multiple point data. The point data includes timeline, location information, data type, data value and feature value.
[0051] Define the timeline dimension as T, the data type dimension as D, and the location information dimension as S to construct a multidimensional space, that is, to construct a three-dimensional coordinate system through the time dimension T, the data type dimension D, and the location information dimension S. By constructing a data cube, the point data in the multidimensional time series plane are associated.
[0052] For easier understanding, the following examples are given for further explanation. Figure 2 , a certain position has three types of data, A, B, and C. Data types A, B, and C respectively construct multidimensional time series planes with the timeline, and the data values of the position on different timelines form coordinate points in the multidimensional time series plane. Construct a multidimensional space, and the constructed multidimensional time series planes are arranged along the location information dimension. Usually, if data types A, B, and C are in the same position, the multidimensional time series planes are overlapped in the multidimensional space. In order to facilitate display and analysis, the multidimensional time series planes are arranged at intervals in the multidimensional space. If data types A, B, and C are in different positions, the multidimensional time series planes are arranged at intervals in the multidimensional space. If data type B and data type C are in the same position and data type A is in a different position, the multidimensional time series planes of data types B and C can overlap or be arranged at intervals in the multidimensional space. The multidimensional time series plane of data type A is arranged at intervals with other multidimensional time series planes in the multidimensional space, and the interval between the multidimensional time series planes of data type B and data type C is smaller than the interval between the multidimensional time series planes of data type B and data type A. The point data forms a data cube in the multi-dimensional space, and the point data are associated by further analyzing the characteristic values in the point data.
[0053] S500, mining IoT data.
[0054] Reshape the point data in the data cube. Using the time dimension as the baseline, traverse all the data in the data cube, calculate the support of the preset item set, compare the support with the preset minimum support threshold, and retain the item set with a support higher than the minimum support threshold. Based on the retained item set, generate all possible two item sets as candidate frequent item sets. Scan all item sets until all candidate frequent item sets are found and no more candidate frequent item sets can be generated.
[0055] Generate association rules through candidate frequent item sets, calculate the confidence and lift of each association rule, compare the confidence with the preset minimum confidence, compare the lift with the preset minimum lift, and retain association rules with confidence greater than or equal to the minimum confidence and lift greater than or equal to the minimum confidence. Perform data prediction based on the retained association rules.
[0056] S600, determining positive deviation data and negative deviation data on the mined data.
[0057] According to the preset judgment conditions, based on the expected effect, expected threshold, pre-constructed model and data change trend, the data validity is judged and the data is divided into positive deviation data and negative deviation data.
[0058] S700, calling the data in the positive deviation database and the negative deviation database to output a visual graph.
[0059] The IoT data and the predictions obtained from data mining are visualized through graphics. Through the visualization of IoT data, the laws, rules, knowledge, patterns, associations, changes, anomalies, trends, relationships, etc. in the IoT data can be discovered to provide support for decision-making.
[0060] The present invention also provides a multi-dimensional time series data cube Internet of Things data mining system, including multiple data acquisition components, which collect and upload corresponding Internet of Things data in the environment. The data acquisition components can be collectors, sensors, etc. The time of different data acquisition components remains consistent; different data acquisition components correspond to unique location information.
[0061] The memory and the processor are communicatively connected, the memory stores instructions that can be executed by the processor, and after the instructions are executed by the processor, the processor can execute the above-mentioned multi-dimensional time series data cube Internet of Things data mining method.
[0062] The display is connected to the processor for displaying output visualization graphics.
[0063] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A multi-dimensional time series data cube IoT data mining method, Features: The steps include: Get IoT data; Extract the timeline, location information, data type and data value from IoT data, reshape and encode IoT data according to preset encoding rules, and perform data cleaning; The IoT data that has completed the shaping encoding is divided into blocks and stored in a pre-built database according to the timeline, location information, data type and data value; Read the IoT data in the database, build a multi-dimensional time series plane based on the timeline through the constructed algorithm model, define different dimensions, and build a multi-dimensional plane so that each multi-dimensional time series plane contains multi-dimensional data distribution; Construct a data cube in a multi-dimensional space to mine IoT data; The mining data generated by the mining is judged, and if it is judged to be positive deviation data, it is stored in the pre-built positive deviation database; If it is determined to be negative deviation data, it will be stored in the pre-built negative deviation database; Call the data in the positive deviation database and the negative deviation database to output visualization graphics; The method to construct a multidimensional plane is: Predefine the timeline dimension as T, the data type dimension as D, and the location information dimension as S; Each data type generates a dimensional plane in the time coordinate, and each location information generates a dimensional plane in the time coordinate, forming a multi-dimensional space; In the multi-dimensional space, according to the time coordinate, a plurality of dimensional planes containing data types and a dimensional plane containing location information are generated; The IoT data in the database is retrieved in a time series manner and stored as point data in a multi-dimensional space. Each dimensional plane contains multiple point data.
2. According to the multi-dimensional time series data cube Internet of Things data mining method of claim 1, Features: The rules for shaping and encoding IoT data are: The 1st to 4th characters are the year, the 5th to 6th characters are the month, the 7th to 8th characters are the day, the 9th to 12th characters are the time, the 13th to 16th characters are the data type, the 17th to 20th characters are the location information, and the 21st to 24th characters are the data value.
3. According to the multi-dimensional time series data cube Internet of Things data mining method of claim 1, Features: Point data includes timeline, location information, data type, data value and feature value.
4. The multi-dimensional time series data cube IoT data mining method according to claim 1, Features: The methods for mining IoT data are: Taking the time dimension as the baseline, traverse the data cube, calculate the support of the preset item set, compare the support with the preset minimum support threshold, and generate two item sets as candidate frequent item sets based on the item set whose support is higher than the minimum support threshold. Repeat until no candidate frequent item sets can be generated. Generate association rules through candidate frequent item sets, calculate the confidence and lift of each association rule, and compare them with the preset minimum confidence and minimum lift, and retain the association rules with confidence greater than or equal to the minimum confidence and lift greater than or equal to the minimum lift.
5. The multi-dimensional time series data cube IoT data mining method according to claim 1, Features: The method for determining the mining data generated by mining is: According to the preset judgment conditions, based on the expected effect, preset thresholds, pre-constructed models and data change trends, the data validity is judged and the data is divided into positive deviation data and negative deviation data.
6. A multi-dimensional time series data cube IoT data mining system, Features: include: Data collection components collect corresponding IoT data in the environment and upload them; A memory and a processor, the memory and the processor are communicatively connected, the memory stores instructions that can be executed by the processor, and after the instructions are executed by the processor, the processor is enabled to execute the multi-dimensional time series data cube Internet of Things data mining method according to any one of claims 1 to 5; The display is connected to the processor for displaying output visualization graphics.
7. The multi-dimensional time series data cube IoT data mining system according to claim 6, Features: When there are multiple data acquisition components, the times of different data acquisition components are kept consistent.
8. The multi-dimensional time series data cube IoT data mining system according to claim 6, Features: When there are multiple data acquisition components, different data acquisition components correspond to unique position information.