Large-scale distributed data analysis processing method and system based on Internet of Things

Abstract

The invention relates to the field of data processing, and discloses a large-scale distributed data analysis processing method and system based on the Internet of Things, which are used for solving the problem that cold and hot data are unreasonably divided during data analysis, and comprises the following steps: collecting Internet of Things data, preprocessing the Internet of Things data, and analyzing the Internet of Things data; the method comprises the steps of preprocessing Internet of Things data, performing data transmission on the preprocessed Internet of Things data, acquiring Internet of Things data information by using a sliding time window method, performing evaluation to obtain a re-division index, judging whether cold and hot data division needs to be performed again or not according to the re-division index, and if the judgment result shows that cold and hot data division needs to be performed again, judging whether cold and hot data division needs to be performed again. If yes, the actual division time interval is calculated according to the re-division index, the cold data and the hot data are stored in a layered mode according to the actual division time interval, high-performance hot storage space is effectively released, waste of hot storage resources is avoided, the storage cost is reduced, and the data management quality and the system performance of the large-scale Internet of Things are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically to a large-scale distributed data analysis and processing method and system based on the Internet of Things. Background Technology

[0002] The rapid development of IoT technology has made large-scale distributed data acquisition and analysis an important research direction. IoT devices collect various data in real time, including environmental, equipment, and user behavior data, through sensors, and transmit the data to cloud or edge computing platforms for analysis. Efficient and accurate data analysis can not only optimize equipment operation and maintenance and improve resource utilization, but also provide reliable support for business decisions.

[0003] In existing technologies, large-scale data analysis in the Internet of Things (IoT) typically employs a fixed hot and cold data management strategy. This involves optimizing data storage and retrieval based on preset time thresholds; for example, data from the most recent three months is considered hot data, while data older than three months is archived as cold data. Hot data is stored in a high-performance database, while cold data is stored in a low-cost storage system. This method can reduce storage costs while ensuring efficient retrieval of recent data.

[0004] However, in the process of implementing the inventive technical solution in the embodiments of this application, it was found that the above-mentioned technology has at least the following technical problems:

[0005] In practical applications, IoT data access patterns are highly dynamic, with significant differences in data access frequency, storage requirements, and analysis methods across different scenarios. For example, in smart grid applications, the demand for electricity data queries increases significantly during peak hours, while the query frequency decreases during off-peak hours. Traditional fixed hot and cold data partitioning methods cannot adapt to this dynamic change, potentially leading to decreased query performance and wasted data storage resources. Summary of the Invention

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides a large-scale distributed data analysis and processing method and system based on the Internet of Things, so as to solve the problems existing in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A large-scale distributed data analysis and processing method based on the Internet of Things (IoT) includes the following steps: Step 1: Data is collected through IoT devices to obtain IoT data; Step 2: The IoT data is preprocessed, including data cleaning, data format conversion, and data compression and filtering, and the preprocessed IoT data is transmitted; Step 3: The transmitted IoT data is received, IoT data information is collected using the sliding time window method, a re-segmentation index is obtained based on the IoT data information, and a judgment is made on whether it is necessary to re-segment the hot and cold data based on the re-segmentation index; Step 4: The initial segmentation time interval is obtained. If it is determined that it is not necessary to re-segment the hot and cold data, the initial segmentation time interval is used as the actual segmentation time interval; if it is determined that it is necessary to re-segment the hot and cold data, the actual segmentation time interval is calculated based on the re-segmentation index; Step 5: Cold data and hot data are stored in layers according to the actual segmentation time interval to obtain stored data; Step 6: Big data analysis and calculation are performed on the stored data to obtain data analysis results; Step 7: The data analysis results are visualized through a visualization platform, and intelligent decision-making is performed.

[0009] Preferably, the step of obtaining the re-division index is as follows: Data access frequency is obtained at each time point within the time window, where each time point is a uniformly selected sampling point within the time window. The data access frequencies are arranged into an access frequency time series. Based on the access frequency time series, the access frequency change coefficient is evaluated using discrete wavelet transform. Query information is obtained at each time point within the time window, including the total number of data queries within the time window and the number of times hot data is hit during the data query. The ratio of the number of times hot data is hit during the data query to the total number of data queries is calculated to obtain the hot storage hit rate. Hot storage resources are obtained at each time point within the time window. The utilization rate of hot storage resources is evaluated using singular value decomposition (SVD) to obtain the hot storage resource utilization rate. A reasonable coefficient for hot storage is calculated based on the hot storage hit rate and the hot storage resource utilization rate. The data acquisition frequency at each time point within the time window is obtained, and K-means clustering is used to cluster the data acquisition frequencies. The data acquisition frequency variation coefficient is calculated based on the clustering results. The access frequency variation coefficient, hot storage reasonable coefficient, and data acquisition frequency variation coefficient are normalized. A reclassification index is obtained by comprehensively evaluating these normalized access frequency variation coefficients, hot storage reasonable coefficients, and data acquisition frequency variation coefficients. The specific steps for obtaining this index are as follows: In the formula, RD represents the re-division index, AF represents the access frequency change coefficient, RS represents the hot storage rationality coefficient, CF represents the data acquisition frequency change coefficient, and a1, a2, and a3 represent the weighting coefficients of the access frequency change coefficient, the hot storage rationality coefficient, and the data acquisition frequency change coefficient, respectively.

[0010] Preferably, the step of obtaining the access frequency change coefficient is as follows: select a wavelet basis function, perform a first-level wavelet decomposition on the access frequency time series through a low-pass filter and a high-pass filter, and split the data into a low-frequency part and a high-frequency part; perform a second-level wavelet decomposition on the low-frequency part to obtain a second-level decomposed low-frequency part and a second-level decomposed high-frequency part; calculate the energy of the high-frequency part based on the high-frequency part and the second-level decomposed high-frequency part, and use it as the access frequency change coefficient.

[0011] Preferably, the hot storage resource utilization rate acquisition step is as follows: setting an embedding dimension, constructing a matrix based on the hot storage resource occupancy rate at each time point within the time window and the embedding dimension; performing singular value decomposition on the matrix to obtain a singular value matrix, obtaining a singular value vector based on the singular value matrix; and calculating the storage resource utilization rate based on the singular value vector.

[0012] Preferably, the step of clustering data collection frequencies using K-means clustering is as follows: Step 3.1: Use data collection frequency as the clustering feature, and take all data collection frequencies within the time window as the dataset, with each data collection frequency in the dataset as a data point; Step 3.2: Use the silhouette coefficient method to determine the number of clusters K in the dataset; Step 3.3: Randomly select K data points in the dataset as initial cluster centers. For each data point, calculate its Euclidean distance to each initial cluster center. For each data point, traverse the K initial cluster centers and assign it to the cluster corresponding to the nearest initial cluster center; Step 3.4: After traversing all data points, obtain the initial clusters. For each initial cluster, calculate the mean of the data points within it to obtain new cluster centers; Step 3.5: Repeat steps 3.3 and 3.4 until the cluster centers no longer change, obtaining the final clusters and final cluster centers.

[0013] Preferably, the step of calculating the data acquisition frequency variation coefficient based on the clustering results is as follows: calculate the weight of each final cluster by the ratio of the number of data points in each final cluster to the total number of data points; and calculate the weight of each final cluster and the final cluster center by weighted summation to obtain the data acquisition frequency variation coefficient.

[0014] Preferably, the step of determining whether to re-divide the hot and cold data based on the re-division index is as follows: compare the re-division index with the division threshold; if the re-division index is greater than or equal to the division threshold, it is determined that the hot and cold data need to be re-divided; if the re-division index is less than the division threshold, it is determined that the hot and cold data do not need to be re-divided.

[0015] Preferably, the step of calculating the actual partitioning time interval based on the re-partitioning index is as follows: calculate the ratio of the partitioning threshold to the re-partitioning index to obtain the adjustment factor; and calculate the actual partitioning time interval by multiplying the adjustment factor with the initial partitioning time interval.

[0016] Preferably, a large-scale distributed data analysis and processing system based on the Internet of Things (IoT) includes: an IoT data acquisition module for acquiring IoT data through IoT devices, obtaining IoT data, and transmitting the IoT data to a data transmission module; a data transmission module for preprocessing the IoT data and transmitting the preprocessed IoT data; a re-division index acquisition module for receiving the transmitted IoT data, collecting data information using a sliding time window method, evaluating a re-division index based on the data information, and determining whether re-division of hot and cold data is necessary based on the re-division index; and an actual division time interval acquisition module for acquiring the initial division time interval and, if determined that re-division is not necessary... If the hot and cold data are re-divided, the initial division time interval is used as the actual division time interval. If it is determined that the hot and cold data need to be re-divided, the actual division time interval is calculated based on the re-division index and transmitted to the hot and cold data division module. The hot and cold data division module stores the cold and hot data in layers according to the actual division time interval, obtaining stored data, and transmits the stored data to the data analysis module. The data analysis module performs big data analysis and calculations based on the stored data, obtains data analysis results, and transmits the data analysis results to the data visualization module. The data visualization module visualizes the data analysis results through a visualization platform and enables intelligent decision-making.

[0017] The technical effects and advantages of this invention are as follows:

[0018] The system collects IoT data, preprocesses it, and then transmits the preprocessed data. It uses a sliding time window method to collect IoT data and evaluates the re-segmentation index. Based on this index, it determines whether a new hot and cold data segmentation is needed. If a new segmentation is required, the actual segmentation time interval is calculated based on the index. Cold and hot data are then stored in tiers according to this actual time interval, effectively freeing up high-performance hot storage space, avoiding waste of hot storage resources, reducing storage costs, and improving the quality and performance of large-scale IoT data management. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a large-scale distributed data analysis and processing method based on the Internet of Things (IoT) is provided for embodiments of this application.

[0020] Figure 2 This is a structural diagram of a large-scale distributed data analysis and processing system based on the Internet of Things, provided for an embodiment of this application. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. In addition, the forms of the various structures described in the following embodiments are merely illustrative. The large-scale distributed data analysis and processing method and system based on the Internet of Things involved in the present invention are not limited to the structures described in the following embodiments. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] This invention provides a large-scale distributed data analysis and processing method based on the Internet of Things, such as... Figure 1 As shown, it includes the following steps:

[0023] Step 1: Collect data through IoT devices to obtain IoT data;

[0024] Step 2: Preprocess the IoT data, including data cleaning, data format conversion, data compression and filtering, and then transmit the preprocessed IoT data.

[0025] Step 3: Receive the transmitted IoT data, determine the distributed storage architecture, collect IoT data information using the sliding time window method, evaluate the IoT data information to obtain the re-division index, and determine whether it is necessary to re-divide the hot and cold data based on the re-division index.

[0026] A sliding time window is a method for dynamically analyzing data over a continuous time period. Its core idea is to continuously slide forward within a fixed-length time window, collecting and processing data from the most recent period in real time. In this embodiment, the sliding time window is used to continuously collect and statistically analyze data from the IoT system, ensuring that the evaluation of the reclassification index is always based on the latest, timely data that reflects the current system operating status.

[0027] Using a sliding time window enables real-time monitoring and trend assessment of dynamic data behavior, avoiding the lag inherent in analysis based on global static data. By limiting the analysis scope to a recent period, the system can respond more sensitively to changes in access patterns and data characteristics, thereby improving the accuracy and timeliness of index reclassification and ensuring that the hot and cold data partitioning strategy can flexibly adapt to current business needs and system load changes.

[0028] In this embodiment, it should be specifically explained that the step of obtaining the re-division index is as follows:

[0029] Data access frequency is obtained at each time point within the time window. The time points are sampling points selected evenly within the time window. The data access frequency is arranged into an access frequency time series. Based on the access frequency time series, the access frequency change coefficient is evaluated by discrete wavelet transform.

[0030] Discrete wavelet transform (DWT) is a mathematical transformation method used to analyze non-stationary signals. It decomposes the original signal into sub-signals of different frequency ranges through multi-scale decomposition to extract its local time-frequency features. In access frequency time series analysis, DWT decomposes the data step-by-step using low-pass and high-pass filters to obtain low-frequency and high-frequency coefficients, respectively. The energy of the high-frequency coefficients can measure the degree of change in access patterns, thus allowing the calculation of access frequency variation coefficients and providing a basis for classifying hot and cold data.

[0031] The query information is obtained at each time point within the time window. The query information is the total number of times data is queried within the time window and the number of times hot data is hit when data is queried. The ratio of the number of times hot data is hit when data is queried to the total number of times data is queried is calculated to obtain the hot storage hit rate.

[0032] The hot storage resource occupancy rate at each time point within the time window is obtained. The hot storage resource occupancy rate is evaluated using the singular value decomposition method to obtain the hot storage resource utilization rate. Singular value decomposition is a commonly used matrix decomposition method that decomposes any real matrix into the product of three matrices. In the evaluation of hot storage resource occupancy rate, the occupancy rate sequence within the time window is constructed into an embedding matrix through a sliding window and then subjected to singular value decomposition. The concentration and effectiveness of resource utilization can be measured by the proportion of the total energy occupied by the first few major singular values. This proportion is the storage resource utilization rate, which reflects whether the resource usage structure of the system in the time dimension is stable and efficient.

[0033] The thermal storage rationality coefficient is calculated based on the thermal storage hit rate and thermal storage resource utilization rate. The specific steps for obtaining this coefficient are as follows:

[0034]

[0035] In the formula, RS represents the hot storage rationality coefficient, ru represents the hot storage resource utilization rate, and qh represents the hot storage hit rate. Based on the harmonic mean method, this formula assesses whether hot storage resources are being used rationally and efficiently, considering both resource occupancy and query hit rate. Only when both are at high levels will the hot storage rationality coefficient approach 1, indicating that hot data is frequently used and allocated sufficient storage resources. Conversely, if either indicator is low, the coefficient will decrease significantly, reflecting a mismatch in hot storage usage. This formula uses the mathematical form of the harmonic mean, a calculation method that emphasizes balance and matching. Compared to the arithmetic mean, the harmonic mean is more sensitive when dealing with two positively correlated but potentially unbalanced indicators, effectively penalizing "one-sided bias" situations and avoiding misjudging overall utilization efficiency due to a high value for a single indicator.

[0036] Obtain the data collection frequency at each time point within the time window, use K-means clustering to cluster the data collection frequencies, and calculate the data collection frequency variation coefficient based on the clustering results;

[0037] K-means clustering is a commonly used unsupervised learning algorithm used to divide data into K clusters, ensuring high similarity among data points within the same cluster and significant differences between different clusters. This method iteratively selects K initial cluster centers, then assigns each data point to the cluster corresponding to its nearest center. The center of each cluster is then updated to the mean of all data points within that cluster, until the cluster centers no longer change significantly. When clustering data collection frequencies at different time points within a time window, K-means can effectively identify the distribution characteristics of different frequency patterns, thus helping to assess whether there are significant fluctuations in collection behavior and providing a basis for calculating the coefficient of variation in data collection frequency.

[0038] The access frequency variation coefficient, hot storage rationality coefficient, and data acquisition frequency variation coefficient are normalized. Based on the normalized access frequency variation coefficient, hot storage rationality coefficient, and data acquisition frequency variation coefficient, a reclassification index is obtained through comprehensive evaluation. The specific steps are as follows:

[0039]

[0040] In the formula, RD represents the re-partitioning index, AF represents the access frequency change coefficient, and the greater the fluctuation in the access frequency of data within the time window, the stronger the system's need to adjust the hot and cold data partitioning. If frequently accessed data suddenly decreases in access, or the access volume of previously unpopular data increases significantly, it will trigger a change in the rationality of the hot and cold data boundary. Timely judgment is needed to determine whether hot and cold data need to be re-partitioned, thereby optimizing the storage structure and query efficiency. RS represents the hot storage rationality coefficient. When the hot storage resource utilization rate and hit rate are highly matched, and the storage and access efficiency of hot data is high, it indicates that the current hot and cold data partitioning is reasonable. At this time, no adjustment is needed, and the urgency of re-partitioning is low, so the re-partitioning index is small. Conversely, when the hot storage rationality coefficient is low, it means that there is a waste of hot storage resources or poor hit efficiency, and the hot and cold partitioning is mismatched. Therefore, the re-partitioning index increases. CF represents the data acquisition frequency change coefficient. When the data acquisition frequency of IoT devices changes significantly, the system's need to adjust the hot and cold data boundary increases accordingly. For example, a sudden increase in the sampling frequency of a certain type of sensor can lead to a surge of new data, potentially overshadowing the existing hot data range and failing to cover the newly added high-value data. Conversely, a decrease in the sampling frequency may result in a waste of hot storage resources. a1 + a2 + a3 = 1, where a1, a2, and a3 represent the weighting coefficients of the access frequency change coefficient, the hot storage rationality coefficient, and the data acquisition frequency change coefficient, respectively. a1, a2, and a3 are obtained through the Analytic Hierarchy Process (AHP), a multi-index decision analysis method. AHP decomposes complex problems into multiple interrelated factors by constructing a hierarchical model. Through expert scoring or empirical judgment, each factor is compared pairwise to construct a judgment matrix, thereby calculating the relative weight of each factor. In this embodiment, AHP is used to determine the weights of various indicators affecting the reclassification index, ensuring a more scientific and reasonable allocation of importance among different indicators in the comprehensive evaluation, thereby improving the accuracy and effectiveness of the hot and cold data classification strategy.

[0041] In this embodiment, it should be specifically explained that the step of obtaining the access frequency change coefficient is as follows:

[0042] Choose an appropriate wavelet basis function. Common wavelet basis functions include Haar wavelet, Daubechies(db) wavelet, Symlet wavelet and Coiflet wavelet.

[0043] The access frequency time series is decomposed into a first-level wavelet decomposition by passing low-pass and high-pass filters, splitting the data into low-frequency and high-frequency components for analysis of changes in different frequency components. Typically, the high-frequency component reflects short-term drastic changes, while the low-frequency component reflects long-term trends.

[0044] A low-pass filter is a signal processing tool used to preserve low-frequency components in a signal while filtering out high-frequency components. In wavelet transform, low-pass filters are used to extract long-term trends in accessed frequency time series, making the data smoother.

[0045] A high-pass filter is a tool used to extract high-frequency components of a signal. It preserves the rapidly changing information of the data while removing smoother low-frequency trends. In wavelet transform, the combination of a high-pass filter and downsampling can generate detail coefficients to represent local short-term fluctuations in the data.

[0046] Since the low-frequency part obtained by the first-level wavelet decomposition still contains a lot of information, the low-frequency part is subjected to a second-level wavelet decomposition to obtain the second-level decomposed low-frequency part and the second-level decomposed high-frequency part.

[0047] Based on the high-frequency component and the high-frequency component of the second-order decomposition, the energy of the high-frequency component is calculated and used as the access frequency variation coefficient to measure the degree of fluctuation of the access frequency. The specific steps for obtaining this coefficient are as follows:

[0048]

[0049] E D Represented as the energy of the high-frequency component, a larger value indicates greater fluctuation in the access frequency, j is the wavelet decomposition level, and D... j [i] represents the i-th high-frequency coefficient of the j-th level decomposition, N j This represents the number of data points in the high-frequency component after the j-th level decomposition.

[0050] In this embodiment, it should be specifically explained that the steps for obtaining the utilization rate of thermal storage resources are as follows:

[0051] Define the embedding dimension, for example, 3. Construct a matrix based on the hot storage resource occupancy rate at each time point within the time window and the embedding dimension. For example, the matrix could be:

[0052]

[0053] In the formula, each row represents the hot storage resource utilization rate at consecutive time points;

[0054] Perform singular value decomposition on matrix R to obtain the singular value matrix, and then obtain the singular value vector [σ1,σ2,σ3,…,σ] from the singular value matrix. d It should be noted that singular value decomposition of a matrix is ​​a prior art technique, and the specific process will not be described in detail in this embodiment;

[0055] Storage resource utilization is calculated based on singular value vectors, specifically obtained as follows:

[0056]

[0057] Where ru represents the hot storage resource utilization rate, σ1 represents the first singular value, and d is the total number of singular values. If the access ratio changes stably and is concentrated, most of the energy will be concentrated in the first singular value σ1, and the utilization coefficient will be close to 1. The singular values ​​are squared before calculation because the singular value matrix is ​​a diagonal matrix. The singular values ​​themselves represent the scale or stretch of the matrix in different principal directions, but the squares of these values ​​are more meaningful, representing the energy of the data in this direction.

[0058] In this embodiment, it should be specifically explained that the steps for clustering the data collection frequency using the K-means clustering method are as follows:

[0059] Step 3.1: Use the data collection frequency as the clustering feature, and use all data collection frequencies within the time window as the dataset, with each data collection frequency in the dataset being a data point;

[0060] Step 3.2: Determine the number of clusters K in the dataset using the silhouette coefficient method;

[0061] The silhouette coefficient method is a technique used to evaluate clustering performance and determine the optimal number of clusters K. This method calculates the silhouette coefficient for each data point, comprehensively measuring its density within a cluster and its separation from its nearest neighbor clusters.

[0062] Step 3.3: Randomly select K data points in the dataset as initial cluster centers. For each data point, calculate its Euclidean distance to each initial cluster center. For each data point, traverse the K initial cluster centers and assign it to the cluster corresponding to the nearest initial cluster center.

[0063] Euclidean distance is a commonly used distance metric that measures the straight-line distance between two points. It reflects the "geometric straight-line distance" between two points in multidimensional space; the smaller the distance, the more similar they are. Euclidean distance is widely used in cluster analysis, classification algorithms, and similarity calculations, and is particularly suitable for clustering decisions based on numerical feature data.

[0064] Step 3.4: After traversing all data points, the initial clusters are obtained. For each initial cluster, the mean of the data points within it is calculated to obtain the new cluster centers.

[0065] Step 3.5: Repeat steps 3.3 and 3.4 until the cluster centers no longer change, and obtain the final clusters and final cluster centers.

[0066] In this embodiment, it should be specifically explained that the step of calculating the data acquisition frequency variation coefficient based on the clustering results is as follows:

[0067] The weight of each final cluster is calculated by dividing the number of data points in each final cluster by the total number of data points.

[0068] The data collection frequency variation coefficient is obtained by weighting and summing the weights of each final cluster with the final cluster center.

[0069] In this embodiment, it should be specifically explained that the step of determining whether to re-divide the hot and cold data based on the re-division index is as follows:

[0070] The re-segmentation index is compared with the segmentation threshold. If the re-segmentation index is greater than or equal to the segmentation threshold, it is determined that the hot and cold data need to be re-segmented; if the re-segmentation index is less than the segmentation threshold, it is determined that the hot and cold data need not be re-segmented. The segmentation threshold is obtained through an adaptive threshold method, which is a method that dynamically adjusts the judgment criteria according to the system's operating status. It is mainly used to avoid the problem of poor adaptability of fixed thresholds in different application scenarios. In this invention, the adaptive threshold method continuously monitors indicators such as data access frequency, query latency, resource utilization, and hot and cold data hit ratio in the IoT system, and analyzes them in conjunction with statistical characteristics within a sliding time window to obtain the optimal segmentation threshold that reflects the current system load and data access behavior in real time. This ensures that the segmentation threshold can self-adjust according to business changes and data characteristics, thereby improving the accuracy and adaptability of hot and cold data segmentation.

[0071] Step 4: Obtain the initial partitioning time interval. If it is determined that the hot and cold data need not be re-partitioned, the initial partitioning time interval is used as the actual partitioning time interval; if it is determined that the hot and cold data need to be re-partitioned, the actual partitioning time interval is calculated based on the re-partitioning index.

[0072] In this embodiment, it should be specifically explained that the step of calculating the actual partitioning time interval based on the re-partitioning index is as follows:

[0073] The adjustment factor is obtained by calculating the ratio between the partitioning threshold and the repartitioning index;

[0074] The actual partitioning time interval is obtained by multiplying the adjustment factor with the initial partitioning time interval. Based on the idea of ​​dynamic control, a quantitative adjustment factor is constructed by introducing the proportional relationship between the partitioning threshold and the actual load change, so that the hot and cold data partitioning mechanism can make accurate responses, thereby achieving dynamic matching between partitioning frequency and data behavior.

[0075] Step 5: Store cold data and hot data separately in layers according to the actual time interval to obtain the stored data;

[0076] Hot data refers to data that is frequently accessed, queried, or updated within a certain time frame, typically exhibiting high real-time performance and business relevance. In IoT systems, hot data often includes the latest sensor data, device operating status, and abnormal alarm records, requiring storage in high-performance, low-latency storage systems to meet the needs of real-time analysis, rapid querying, and decision response.

[0077] Cold data refers to data that is accessed less frequently and has less stringent timeliness requirements at the current stage. It is mainly used for long-term data accumulation scenarios such as historical archiving, trend analysis, and model training. In IoT systems, cold data typically includes past collection logs, device operation records, and historical analysis results. It can be stored in a lower-cost but higher-capacity storage system to reduce system resource consumption and storage costs while still retaining necessary access capabilities.

[0078] Step 6: Perform big data analysis and calculations based on the stored data to obtain the data analysis results;

[0079] Step 7: Visualize the data analysis results through a visualization platform and make intelligent decisions, including predictive maintenance, anomaly detection, and business optimization.

[0080] In this embodiment, it is necessary to specifically describe a large-scale distributed data analysis and processing system based on the Internet of Things, such as... Figure 2 As shown, the system includes:

[0081] The IoT data acquisition module is used to acquire IoT data through IoT devices, obtain IoT data, and transmit the IoT data to the data transmission module.

[0082] The data transmission module is used to preprocess IoT data and transmit the preprocessed IoT data.

[0083] The reclassification index acquisition module is used to receive transmitted IoT data, collect data information using the sliding time window method, evaluate the reclassification index based on the data information, and determine whether it is necessary to reclassify the hot and cold data based on the reclassification index.

[0084] The actual partitioning time interval acquisition module is used to obtain the initial partitioning time interval. If it is determined that there is no need to re-partition the hot and cold data, the initial partitioning time interval is used as the actual partitioning time interval. If it is determined that there is a need to re-partition the hot and cold data, the actual partitioning time interval is calculated based on the re-partitioning index and transmitted to the hot and cold data partitioning module.

[0085] The cold and hot data partitioning module stores cold data and hot data in layers according to the actual partitioning time interval, obtains the stored data, and transmits the stored data to the data analysis module.

[0086] The data analysis module is used to perform big data analysis and calculations based on stored data, obtain data analysis results, and transmit the data analysis results to the data visualization module;

[0087] The data visualization module is used to visualize data analysis results through a visualization platform and to make intelligent decisions.

[0088] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0089] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A large-scale distributed data analysis and processing method based on the Internet of Things, characterized in that, Includes the following steps: Step 1: Collect data through IoT devices to obtain IoT data; Step 2: Preprocess the IoT data, including data cleaning, data format conversion, data compression and filtering, and then transmit the preprocessed IoT data. Step 3: Receive the transmitted IoT data, collect IoT data information using the sliding time window method, evaluate the IoT data information to obtain the reclassification index, and determine whether it is necessary to reclassify the hot and cold data based on the reclassification index. Step 4: Obtain the initial partitioning time interval. If it is determined that the hot and cold data need not be re-partitioned, the initial partitioning time interval is used as the actual partitioning time interval; if it is determined that the hot and cold data need to be re-partitioned, the actual partitioning time interval is calculated based on the re-partitioning index. Step 5: Store cold data and hot data separately in layers according to the actual time interval to obtain the stored data; Step 6: Perform big data analysis and calculations based on the stored data to obtain the data analysis results; Step 7: Visualize the data analysis results through a visualization platform and make intelligent decisions.

2. The large-scale distributed data analysis and processing method based on the Internet of Things according to claim 1, characterized in that: The steps for obtaining the reclassification index are as follows: Data access frequency is obtained at each time point within the time window. The time points are sampling points selected evenly within the time window. The data access frequency is arranged into an access frequency time series. Based on the access frequency time series, the access frequency change coefficient is evaluated by discrete wavelet transform. The query information is obtained at each time point within the time window. The query information includes the total number of data queries within the time window and the number of times hot data is hit during the data query. The ratio of the number of hot data hits during the data query to the total number of data queries is calculated to obtain the hot storage hit rate. The hot storage resource occupancy rate at each time point within the time window is obtained, and the hot storage resource occupancy rate is evaluated using the singular value decomposition method to obtain the hot storage resource utilization rate. The thermal storage rationality coefficient is calculated based on the thermal storage hit rate and thermal storage resource utilization rate. Obtain the data collection frequency at each time point within the time window, use K-means clustering to cluster the data collection frequencies, and calculate the data collection frequency variation coefficient based on the clustering results; The access frequency variation coefficient, hot storage rationality coefficient, and data acquisition frequency variation coefficient are normalized. Based on the normalized access frequency variation coefficient, hot storage rationality coefficient, and data acquisition frequency variation coefficient, a reclassification index is obtained through comprehensive evaluation. The specific steps are as follows: In the formula, RD represents the re-division index, AF represents the access frequency change coefficient, RS represents the hot storage rationality coefficient, CF represents the data acquisition frequency change coefficient, and a1, a2, and a3 represent the weighting coefficients of the access frequency change coefficient, the hot storage rationality coefficient, and the data acquisition frequency change coefficient, respectively.

3. The large-scale distributed data analysis and processing method based on the Internet of Things according to claim 2, characterized in that, The steps for obtaining the access frequency variation coefficient are as follows: By selecting wavelet basis functions, the access frequency time series is decomposed into low-frequency and high-frequency components by passing low-pass and high-pass filters. The low-frequency component is subjected to a second-level wavelet decomposition to obtain the second-level decomposed low-frequency component and the second-level decomposed high-frequency component. Based on the high-frequency component and the second-level decomposition of the high-frequency component, the energy of the high-frequency component is calculated and used as the access frequency variation coefficient.

4. The large-scale distributed data analysis and processing method based on the Internet of Things according to claim 2, characterized in that, The steps for obtaining the utilization rate of thermal storage resources are as follows: Define the embedding dimension and construct a matrix based on the hot storage resource occupancy rate at each time point within the time window and the embedding dimension. Perform singular value decomposition on the matrix to obtain the singular value matrix, and then obtain the singular value vector based on the singular value matrix; Storage resource utilization is calculated based on singular value vectors.

5. The large-scale distributed data analysis and processing method based on the Internet of Things according to claim 2, characterized in that: The steps for clustering data collection frequencies using K-means clustering are as follows: Step 3.1: Use the data collection frequency as the clustering feature, and use all data collection frequencies within the time window as the dataset, with each data collection frequency in the dataset being a data point; Step 3.2: Determine the number of clusters K in the dataset using the silhouette coefficient method; Step 3.3: Randomly select K data points in the dataset as initial cluster centers. For each data point, calculate its Euclidean distance to each initial cluster center. For each data point, traverse the K initial cluster centers and assign it to the cluster corresponding to the nearest initial cluster center. Step 3.4: After traversing all data points, the initial clusters are obtained. For each initial cluster, the mean of the data points within it is calculated to obtain the new cluster centers. Step 3.5: Repeat steps 3.3 and 3.4 until the cluster centers no longer change, and obtain the final clusters and final cluster centers.

6. The large-scale distributed data analysis and processing method based on the Internet of Things according to claim 2, characterized in that: The step of calculating the data acquisition frequency variation coefficient based on the clustering results is as follows: The weight of each final cluster is calculated by dividing the number of data points in each final cluster by the total number of data points. The data collection frequency variation coefficient is obtained by weighting and summing the weights of each final cluster with the final cluster center.

7. The large-scale distributed data analysis and processing method based on the Internet of Things according to claim 1, characterized in that: The step of determining whether the hot and cold data need to be reclassified based on the reclassification index is as follows: The re-segmentation index is compared with the segmentation threshold. If the re-segmentation index is greater than or equal to the segmentation threshold, it is determined that the hot and cold data need to be re-segmented; if the re-segmentation index is less than the segmentation threshold, it is determined that the hot and cold data do not need to be re-segmented.

8. The large-scale distributed data analysis and processing method based on the Internet of Things according to claim 1, characterized in that: The step of calculating the actual partitioning time interval based on the re-partitioning index is as follows: The adjustment factor is obtained by calculating the ratio between the partitioning threshold and the repartitioning index; The actual partitioning time interval is obtained by multiplying the adjustment factor by the initial partitioning time interval.

9. A large-scale distributed data analysis and processing system based on the Internet of Things (IoT), used to implement the large-scale distributed data analysis and processing method based on the Internet of Things as described in any one of claims 1-8, characterized in that: The system includes: The IoT data acquisition module is used to acquire IoT data through IoT devices, obtain IoT data, and transmit the IoT data to the data transmission module. The data transmission module is used to preprocess IoT data and transmit the preprocessed IoT data. The reclassification index acquisition module is used to receive transmitted IoT data, collect data information using the sliding time window method, evaluate the reclassification index based on the data information, and determine whether it is necessary to reclassify the hot and cold data based on the reclassification index. The actual partitioning time interval acquisition module is used to obtain the initial partitioning time interval. If it is determined that there is no need to re-partition the hot and cold data, the initial partitioning time interval is used as the actual partitioning time interval. If it is determined that there is a need to re-partition the hot and cold data, the actual partitioning time interval is calculated based on the re-partitioning index and transmitted to the hot and cold data partitioning module. The cold and hot data partitioning module stores cold data and hot data in layers according to the actual partitioning time interval, obtains the stored data, and transmits the stored data to the data analysis module. The data analysis module is used to perform big data analysis and calculations based on stored data, obtain data analysis results, and transmit the data analysis results to the data visualization module; The data visualization module is used to visualize data analysis results through a visualization platform and to make intelligent decisions.

Images