A method for processing data of Internet of Things

Abstract

The present invention provides an Internet of Things data processing method, which relates to the field of data processing technology, including generating a coding model based on a historical data point set of an IoT sensor and defining an initial threshold, wherein the initial threshold is used to judge whether the historical data points in the historical data point set are abnormal; acquiring real-time data points of the IoT sensor and inputting them into the coding model for data reconstruction to obtain a reconstruction error, and calculating a reconstruction standard deviation and a reconstruction mean based on the reconstruction error; using a sliding window update mechanism to fill the reconstruction error into a preset sliding window of a predefined time length in chronological order; updating the initial threshold based on the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to obtain a dynamic threshold. The present invention significantly improves the quality and reliability of Internet of Things data by performing data cleaning based on the dynamic threshold.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method for processing Internet of Things data. Background Art

[0002] With the rapid development of Internet of Things (IoT) technology, IoT sensors have been widely used in many fields such as smart industry, smart grid, smart transportation, and environmental monitoring. These sensors continuously generate massive amounts of data, which are not only an important reflection of the system's operating status, but also the basis for intelligent decision-making and optimization. However, due to the complexity and uncertainty of the IoT environment, sensor data often contains a large amount of noise, outliers, and redundant information, which seriously affect the quality of the data and, in turn, the accuracy of decision-making and control based on these data.

[0003] In existing IoT data processing methods, static thresholds are usually used to determine whether data is abnormal. However, static thresholds cannot adapt to the dynamic characteristics of data in IoT environments. Especially in complex and changeable industrial or environmental scenarios, the volatility and uncertainty of data make it difficult for static threshold methods to effectively identify abnormal data points. In addition, traditional data processing methods often lack an in-depth understanding of the intrinsic structure and characteristics of data, resulting in low efficiency in processing massive, polymorphic, and dynamic IoT data, and it is difficult to ensure the accuracy and reliability of data.

[0004] Therefore, it is necessary to provide an Internet of Things data processing method to solve the above technical problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides an Internet of Things data processing method, which screens out abnormal data points based on dynamic thresholds and performs abnormal data cleaning, thereby significantly improving the quality and reliability of Internet of Things data.

[0006] The present invention provides a method for processing Internet of Things data, the processing method comprising the following steps:

[0007] Generate a coding model based on the historical data point set of the IoT sensor and define an initial threshold, wherein the initial threshold is used to determine whether a historical data point in the historical data point set is abnormal;

[0008] Acquire the real-time data points of the IoT sensor and input them into the encoding and decoding model to reconstruct the data, obtain a reconstruction error, and calculate a reconstruction standard deviation and a reconstruction mean based on the reconstruction error;

[0009] Using a sliding window update mechanism, the reconstruction error is filled into a preset sliding window of a predefined time length in chronological order;

[0010] The initial threshold is updated based on the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to obtain a dynamic threshold;

[0011] The reconstruction error corresponding to each real-time data point is screened based on the dynamic threshold to obtain abnormal data points, and the abnormal data points are cleaned.

[0012] Preferably, the generating of the encoding and decoding model based on the historical data point set of the IoT sensor and defining the initial threshold value includes:

[0013] Standardizing the historical data point set to obtain training data;

[0014] Performing feature extraction on the training data to obtain a feature set;

[0015] Model training is performed based on the feature set to obtain a codec model, wherein the codec model includes an encoder for performing low-dimensional representation of data and a decoder for reconstructing the low-dimensional representation.

[0016] Preferably, the generating a coding model based on the historical data point set of the IoT sensor and defining an initial threshold value further includes:

[0017] In the process of constructing the encoding and decoding model, a historical reconstruction error set consisting of multiple historical reconstruction error combinations including initial standard deviations and initial means is obtained at the same time, and an initial threshold is calculated according to the initial standard deviation and the initial mean, wherein the calculation formula of the initial threshold is: ;

[0018] in, is the initial threshold, is the initial mean, is the initial standard deviation, is a predefined constant.

[0019] Preferably, the acquiring of the real-time data points of the IoT sensor and inputting the data into the encoding and decoding model for data reconstruction to obtain a reconstruction error, and calculating a reconstruction standard deviation and a reconstruction mean based on the reconstruction error, comprises:

[0020] Read the real-time data points output by the IoT sensor and perform standardization processing;

[0021] The real-time data points after the standardization process are encoded by the encoder of the encoding and decoding model to obtain a low-dimensional representation of the real-time data points;

[0022] Reconstructing the low-dimensional representation of the real-time data point through the decoder of the encoding and decoding model to obtain a reconstructed data point;

[0023] The difference between the real-time data point and the reconstructed data point is measured based on the Euclidean distance to obtain a reconstruction error, and a reconstruction standard deviation and a reconstruction mean are calculated based on the reconstruction error.

[0024] Preferably, the step of filling the reconstruction error into a preset sliding window of a predefined time length in chronological order by using a sliding window update mechanism comprises:

[0025] Organize the reconstruction errors by sorting by timestamps to obtain a list of reconstruction errors arranged in chronological order;

[0026] Preliminarily filling the preset sliding window with the reconstruction error list arranged in chronological order to obtain a preset sliding window containing a preset number of reconstruction errors;

[0027] The reconstruction error in the preset sliding window after preliminary filling is dynamically updated based on the sliding window update mechanism to keep the reconstruction error in the preset sliding window up to date.

[0028] Preferably, the updating of the initial threshold based on the reconstruction standard deviation and the reconstruction mean of all reconstruction errors in the preset sliding window to obtain a dynamic threshold comprises:

[0029] Adding the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to the historical reconstruction error set;

[0030] The real-time mean and real-time standard deviation are calculated by all the initial standard deviations and initial means in the historical reconstruction error set and the reconstruction standard deviations and reconstruction means of all reconstruction errors in the preset sliding window, wherein the calculation formula of the real-time mean is: ;

[0031] in, Real-time average, and are the initial mean and the reconstructed mean, respectively. and are the first weight of the initial mean and the second weight of the reconstructed mean respectively;

[0032] The calculation formula of the real-time standard deviation is: ;

[0033] in, and are the number of historical data points and real-time data points in the preset sliding window, respectively. and are the initial standard deviation and the reconstructed standard deviation respectively;

[0034] The dynamic threshold is calculated based on the real-time standard deviation and the real-time mean, wherein the calculation formula of the dynamic threshold is: ;

[0035] in, is the dynamic threshold, is the real-time mean, is the real-time standard deviation, is a predefined constant.

[0036] Preferably, the values ​​of the first weight and the second weight are 0.3 and 0.7 respectively.

[0037] Preferably, screening the reconstruction error corresponding to each real-time data point based on the dynamic threshold to obtain abnormal data points, and cleaning the abnormal data points includes:

[0038] The dynamic threshold is compared with the reconstruction error of the real-time data point. If the reconstruction error exceeds the dynamic threshold, the real-time data point corresponding to the reconstruction error is identified as an abnormal data point, and the abnormal data point is removed.

[0039] Compared with the related art, the IoT data processing method provided by the present invention has the following beneficial effects:

[0040] The present invention generates a coding model based on a historical data point set of an IoT sensor and defines an initial threshold, wherein the initial threshold is used to determine whether a historical data point in the historical data point set is abnormal; the real-time data points of the IoT sensor are obtained and input into the coding model for data reconstruction to obtain a reconstruction error, and a reconstruction standard deviation and a reconstruction mean are calculated based on the reconstruction error; the reconstruction error is filled into a preset sliding window of a predefined time length in chronological order by using a sliding window update mechanism; the initial threshold is updated based on the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to obtain a dynamic threshold. The present invention significantly improves the quality and reliability of IoT data by performing data cleaning based on the dynamic threshold. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 The present invention provides a flowchart of a method for processing Internet of Things data. DETAILED DESCRIPTION

[0042] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0043] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the various operations (or steps) as being processed sequentially, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the various operations can be rearranged. The processing can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The processing can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0044] The present invention provides a method for processing Internet of Things data, the processing method comprising the following steps:

[0045] S1: Generate a coding model based on a historical data point set of an IoT sensor and define an initial threshold, wherein the initial threshold is used to determine whether a historical data point in the historical data point set is abnormal.

[0046] In this embodiment, a set of historical data points within a period of time is collected as training samples; then, an autoencoder is used to train these data points to build an encoding and decoding model that can effectively encode and decode data. At the same time, an initial threshold is set based on the statistical analysis of historical data, and the threshold is used to preliminarily judge the abnormality of the data point.

[0047] The model learns the distribution characteristics of normal data, laying the foundation for subsequent anomaly detection and ensuring that the model training fully reflects the characteristics of the actual data. At the same time, the initial threshold is used to provide a reference benchmark when the model has not fully adapted to environmental changes, which helps to avoid too many false positives or negatives during the model initialization stage.

[0048] S2: Acquire the real-time data points of the IoT sensor and input them into the encoding and decoding model to reconstruct the data, obtain a reconstruction error, and calculate a reconstruction standard deviation and a reconstruction mean based on the reconstruction error.

[0049] In this embodiment, IoT sensor data points are read in real time and input into a trained encoding and decoding model. The model outputs reconstructed data points, and then the difference between the original data points and the reconstructed data points, i.e., the reconstruction error, is calculated, thereby quantifying the deviation between the real-time data points and the expected data of the model, providing a quantitative basis for subsequent anomaly detection.

[0050] S3: Using a sliding window update mechanism, the reconstruction error is filled into a preset sliding window of a predefined time length in chronological order.

[0051] In this embodiment, a time window of fixed length is created. Whenever a new reconstruction error is generated, it is added to the window, and the error that enters the window earliest is removed to keep the number of errors in the window constant.

[0052] Through the sliding window mechanism, dynamic monitoring of the reconstruction error of data points in the recent period of time is achieved, so that anomaly detection can better adapt to environmental changes, thereby improving the dynamic adaptability and accuracy of anomaly detection and reducing misjudgments caused by environmental fluctuations.

[0053] S4: The initial threshold is updated based on the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to obtain a dynamic threshold.

[0054] In this embodiment, the mean and standard deviation of the reconstruction error in the sliding window are calculated, and the initial threshold is adjusted based on these statistics to form a dynamic threshold. Such a dynamic threshold can more accurately reflect the statistical characteristics of the current data stream, reduce the false positive rate and the false negative rate, and can adapt to the long-term changes in data distribution, thereby improving the robustness and stability of anomaly detection.

[0055] S5: Screening the reconstruction error corresponding to each real-time data point based on the dynamic threshold to obtain abnormal data points, and cleaning the abnormal data points.

[0056] In this embodiment, the reconstruction error of each real-time data point is compared with the dynamic threshold, and the data points that exceed the threshold range are marked as abnormal data points. Subsequently, a data cleaning algorithm is applied to remove these abnormal points, ensuring that the data set only contains data that conforms to the expected pattern, thereby improving the quality and efficiency of subsequent data analysis. At the same time, it also avoids the impact of abnormal data on subsequent analysis, ensuring the accuracy and effectiveness of data analysis.

[0057] Specifically, step S1 includes the following steps:

[0058] S101: Standardize the historical data point set to obtain training data.

[0059] In this embodiment, historical data points are collected over a period of time, and these data points may come from various sensors of IoT devices, such as temperature, humidity, vibration, etc. In order to ensure the effectiveness and consistency of model training, these data need to be preprocessed, including missing value filling, outlier processing, and standardization (such as normalization or Z-score standardization) so that the model can better learn the intrinsic structure of the data.

[0060] S102: Extract features from the training data to obtain a feature set.

[0061] In this embodiment, the most representative and discriminative features are extracted by performing feature engineering on the standardized data. These features include statistical features and frequency domain features of the time series, so as to reduce redundant information and improve the training efficiency and prediction performance of the model.

[0062] S103: Performing model training based on the feature set to obtain a codec model, wherein the codec model includes an encoder for performing low-dimensional representation of data and a decoder for reconstructing the low-dimensional representation.

[0063] In this embodiment, the feature set is used to train the autoencoder model. The encoder part of the model is responsible for converting the input data into a compact low-dimensional representation, while the decoder part attempts to reconstruct the original data from this low-dimensional representation. The training process is completed by minimizing the reconstruction error, ensuring that the model can learn the key features and potential structure of the data.

[0064] S104: During the construction of the encoding and decoding model, a historical reconstruction error set consisting of a plurality of historical reconstruction error combinations including an initial standard deviation and an initial mean is simultaneously obtained, and an initial threshold is calculated according to the initial standard deviation and the initial mean, wherein the calculation formula of the initial threshold is: ;

[0065] in, is the initial threshold, is the initial mean, is the initial standard deviation, is a predefined constant.

[0066] In this embodiment, during model training, the historical reconstruction errors of each historical data point are recorded. These historical reconstruction errors reflect the ability of the model to reconstruct data. By performing statistical analysis on these historical reconstruction errors, the initial mean and initial standard deviation can be calculated, and then the initial threshold is determined by multiplying a predefined constant by the standard deviation plus the mean. This threshold is used to preliminarily identify abnormal data points.

[0067] Specifically, step S2 includes the following steps:

[0068] S201: Read the real-time data points output by the IoT sensor and perform standardization processing.

[0069] In this embodiment, the raw data points output by the sensor are read regularly or continuously through the IoT platform or data acquisition software. These data points may be transmitted in JSON, CSV or binary format and need to be parsed into processable numerical data. Subsequently, the real-time data points are standardized, usually using Z-score standardization, that is, each data point is subtracted from the mean of the historical data and then divided by the standard deviation of the historical data to ensure that the real-time data points have the same scale as the training data. This step ensures that the model can correctly process the real-time data without being affected by the dimension or magnitude.

[0070] S202: Encode the normalized real-time data points through the encoder of the encoding and decoding model to obtain a low-dimensional representation of the real-time data points.

[0071] In this embodiment, the normalized real-time data points are input into the encoder part of the encoding and decoding model. The encoder is a multi-layer neural network that converts high-dimensional data points into a low-dimensional representation, which usually contains the key features of the data points. The encoding process involves forward propagation, where the data points pass through the neurons of the encoder, and the output of each layer is used as the input of the next layer until a compact low-dimensional vector is obtained, which is the low-dimensional representation of the real-time data point.

[0072] S203: Reconstructing the low-dimensional representation of the real-time data point through the decoder of the encoding and decoding model to obtain the reconstructed data point.

[0073] In this embodiment, the decoder part is used to reversely convert the obtained low-dimensional representation back to the original data space. The decoder is also a multi-layer neural network, and its structure is usually symmetrical to the encoder, but in the opposite direction. The low-dimensional representation is used as the input of the decoder, and the dimension of the data is gradually increased through a series of neural network layers until a data point with the same dimension as the original input is reconstructed. This output data point is the reconstructed data point.

[0074] S204: Measuring the difference between the real-time data point and the reconstructed data point based on the Euclidean distance to obtain a reconstruction error, and calculating a reconstruction standard deviation and a reconstruction mean based on the reconstruction error.

[0075] In this embodiment, the Euclidean distance between the real-time data point and the reconstructed data point is calculated, which is achieved by calculating the square root of the sum of the squares of the point-to-point distances between the two vectors. For each real-time data point, there will be a corresponding reconstruction error, which is a measure of the difference between the original data point and the reconstructed data point. Then, the reconstruction errors of all real-time data points are statistically analyzed, and the mean (reconstruction mean) and standard deviation (reconstruction standard deviation) of these errors are calculated. The reconstruction mean reflects the average error of the model reconstructed data, while the reconstruction standard deviation indicates the dispersion of the error. These statistics will be used for subsequent dynamic threshold updates.

[0076] Specifically, step S3 includes the following steps:

[0077] S301: Organize the reconstruction errors by sorting by timestamp to obtain a reconstruction error list arranged in chronological order.

[0078] In this embodiment, the reconstruction errors of all real-time data points are collected and sorted according to the timestamp of each data point. Specifically, a list or array is created, in which each element is a key-value pair containing a timestamp and the corresponding reconstruction error. The sorting process ensures that the data is arranged in the order of arrival time, which is crucial for the subsequent sliding window mechanism.

[0079] S302: Preliminarily filling a preset sliding window with a list of reconstruction errors arranged in chronological order to obtain a preset sliding window containing a preset number of reconstruction errors.

[0080] In this embodiment, a sliding window of a preset length is initialized. The size of this window is usually determined based on historical data analysis and represents the number of data points in a period of time. The first preset number of errors in the sorted reconstructed error list are filled into the sliding window. The length of the window should be large enough to capture the short-term trend of the data stream, but not too large to cause response delays.

[0081] S303: dynamically updating the reconstruction error in the preset sliding window after preliminary filling based on the sliding window update mechanism to keep the reconstruction error in the preset sliding window up to date.

[0082] In this embodiment, as new real-time data points arrive, their reconstruction errors are calculated and added to the tail of the preset sliding window. At the same time, the head of the preset sliding window removes the oldest reconstruction error to keep the number of reconstruction errors in the window constant. This means that the data in the window always reflects the system status in the most recent period of time. The update mechanism of the sliding window ensures that the dynamic threshold can be adjusted according to the latest data stream, improving the real-time and accuracy of anomaly detection.

[0083] Specifically, step S4 includes the following steps:

[0084] S401: adding the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to the historical reconstruction error set.

[0085] In this embodiment, the reconstruction standard deviation and reconstruction mean of all reconstruction errors are collected from the preset sliding window, and these statistics are merged with the historical reconstruction error set to form a comprehensive data set. This step ensures that the statistical characteristics of both new data and historical data are taken into account, providing comprehensive statistical information for the calculation of dynamic thresholds.

[0086] S402: Calculate the real-time mean and real-time standard deviation by using all initial standard deviations and initial means in the historical reconstruction error set and the reconstruction standard deviations and reconstruction means of all reconstruction errors in the preset sliding window, wherein the calculation formula of the real-time mean is: ;

[0087] in, Real-time average, and are the initial mean and the reconstructed mean, respectively. and are the first weight of the initial mean and the second weight of the reconstructed mean respectively;

[0088] Specifically, the values ​​of the first weight and the second weight are 0.3 and 0.7 respectively.

[0089] In this embodiment, in the process of calculating the real-time mean and the real-time standard deviation, the initial mean and standard deviation of the historical data (corresponding to the first weight respectively) are given less weight (0.3), while the reconstructed mean and standard deviation of the real-time data (i.e., the data in the sliding window) (corresponding to the second weight respectively) are given more weight (0.7). By giving the real-time data a greater weight, the real-time mean can be closer to the actual state of the current data stream. At the same time, the mean of the historical data still makes a certain contribution to the result, thereby ensuring the stability of the real-time mean and preventing drastic fluctuations caused by individual outliers.

[0090] The calculation formula of the real-time standard deviation is: ;

[0091] in, and are the number of historical data points and real-time data points in the preset sliding window, respectively. and are the initial standard deviation and the reconstructed standard deviation respectively.

[0092] The calculation of the real-time standard deviation comprehensively considers the standard deviation of historical data and real-time data. By adjusting the weights of each part in the numerator (determined by the number of data points), it ensures that the real-time standard deviation reflects both the stability of historical data and the dynamics of real-time data. The calculation method of the real-time standard deviation ensures that the dynamic adjustment of the anomaly detection threshold can take into account the long-term trend of historical data and the short-term fluctuations of real-time data, improves the flexibility and accuracy of threshold setting, and reduces the risk of false alarms and missed alarms.

[0093] S403: Calculate a dynamic threshold based on the real-time standard deviation and the real-time mean, wherein the calculation formula of the dynamic threshold is: ;

[0094] in, is the dynamic threshold, is the real-time mean, is the real-time standard deviation, is a predefined constant.

[0095] In this embodiment, It is usually set to 3 or higher to ensure that the threshold can effectively distinguish normal data from abnormal data. By substituting the real-time mean and real-time standard deviation into the formula, a dynamically updated threshold can be obtained, which can self-adjust according to the statistical characteristics of the data stream to improve the accuracy and adaptability of anomaly detection.

[0096] Specifically, step S5 includes the following steps:

[0097] The dynamic threshold is compared with the reconstruction error of the real-time data point. If the reconstruction error exceeds the dynamic threshold, the real-time data point corresponding to the reconstruction error is identified as an abnormal data point, and the abnormal data point is removed.

[0098] In this embodiment, the reconstruction error of each real-time data point is compared with a dynamic threshold. If the reconstruction error exceeds the dynamic threshold, it indicates that the data point deviates significantly from the normal behavior pattern expected by the model, and the data point is identified as an abnormal data point. Anomaly detection is based on the difference measure between the real-time data point and the model-reconstructed data point, i.e., the reconstruction error. By comparing it with the dynamic threshold, possible abnormal situations can be identified immediately.

[0099] Once an abnormal data point is detected, it is immediately identified and its status is marked in the data stream. This usually involves attaching an abnormal flag to the data record or storing it in a list of abnormal data points for subsequent processing. Recording information such as the timestamp, original data value, reconstruction error, and anomaly type of the abnormal data point is crucial for analyzing the cause of the anomaly and improving the system.

[0100] The data points identified as abnormal are cleaned. The cleaning process specifically involves deleting the abnormal data points, that is, directly removing them from the data stream.

[0101] The working principle of the Internet of Things data processing method provided by the present invention is as follows: by introducing a coding and decoding model constructed based on a historical data point set of an IoT sensor and combining it with a dynamic threshold adjustment strategy, efficient and accurate processing of Internet of Things data is achieved, which not only significantly improves the recognition accuracy of abnormal data points, but also effectively reduces the false alarm rate and missed alarm rate, thereby ensuring the reliability and effectiveness of data cleaning.

[0102] Specifically, through the reconstruction process of the encoding and decoding model, the intrinsic structure and characteristics of the data are deeply excavated, so that the normal fluctuation range of the data can be more accurately evaluated, providing a scientific basis for the setting of dynamic thresholds. The dynamic threshold can respond to the changing trend of the data in real time, and can maintain a high anomaly detection capability even in the case of large data fluctuations.

[0103] In addition, the application of the sliding window update mechanism further enhances the method's ability to process time series data, enabling it to dynamically adjust the threshold in a continuous data stream to ensure timely and accurate judgment of the latest data. This not only improves the real-time nature of data processing, but also ensures the timeliness and accuracy of the data.

[0104] Ultimately, the quality of IoT data processed by this application has been significantly improved, and noise and abnormal values ​​have been effectively removed, providing more reliable and high-quality data support for subsequent data analysis, intelligent decision-making, etc. This improvement in data processing capabilities will directly promote the in-depth application and development of IoT technology in various fields, bringing more intelligent and efficient solutions and user experience.

[0105] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0106] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0107] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A method for processing Internet of Things data, characterized in that: The processing method comprises the following steps: Generate a coding model based on the historical data point set of the IoT sensor and define an initial threshold, wherein the initial threshold is used to determine whether a historical data point in the historical data point set is abnormal; Acquire the real-time data points of the IoT sensor and input them into the encoding and decoding model to reconstruct the data, obtain a reconstruction error, and calculate a reconstruction standard deviation and a reconstruction mean based on the reconstruction error; Using a sliding window update mechanism, the reconstruction error is filled into a preset sliding window of a predefined time length in chronological order; The initial threshold is updated based on the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to obtain a dynamic threshold; This step specifically includes: Adding the reconstruction standard deviation and reconstruction mean of all reconstruction errors in the preset sliding window to the historical reconstruction error set; The real-time mean and real-time standard deviation are calculated by all the initial standard deviations and initial means in the historical reconstruction error set and the reconstruction standard deviations and reconstruction means of all reconstruction errors in the preset sliding window, wherein the calculation formula of the real-time mean is: ; in, Real-time average, and are the initial mean and the reconstructed mean, respectively. and are the first weight of the initial mean and the second weight of the reconstructed mean respectively; The calculation formula of the real-time standard deviation is: ； in, and are the number of historical data points and real-time data points in the preset sliding window, respectively. and are the initial standard deviation and the reconstructed standard deviation respectively; The dynamic threshold is calculated based on the real-time standard deviation and the real-time mean, wherein the calculation formula of the dynamic threshold is: ; in, is the dynamic threshold, is the real-time mean, is the real-time standard deviation, is a predefined constant; The reconstruction error corresponding to each real-time data point is screened based on the dynamic threshold to obtain abnormal data points, and the abnormal data points are cleaned.

2. The method for processing Internet of Things data according to claim 1, characterized in that: The method of generating a codec model based on a historical data point set of an IoT sensor and defining an initial threshold value includes: Standardizing the historical data point set to obtain training data; Performing feature extraction on the training data to obtain a feature set; Model training is performed based on the feature set to obtain a codec model, wherein the codec model includes an encoder for performing low-dimensional representation of data and a decoder for reconstructing the low-dimensional representation.

3. The method for processing Internet of Things data according to claim 2, characterized in that: The method generates a codec model based on a historical data point set of an IoT sensor and defines an initial threshold, and further includes: In the process of constructing the encoding and decoding model, a historical reconstruction error set consisting of multiple historical reconstruction error combinations including initial standard deviations and initial means is obtained at the same time, and an initial threshold is calculated according to the initial standard deviation and the initial mean, wherein the calculation formula of the initial threshold is: ; in, is the initial threshold, is the initial mean, is the initial standard deviation, is a predefined constant.

4. The method for processing Internet of Things data according to claim 3, characterized in that: The real-time data points of the IoT sensor are obtained and input into the encoding and decoding model for data reconstruction to obtain a reconstruction error, and a reconstruction standard deviation and a reconstruction mean are calculated based on the reconstruction error, including: Read the real-time data points output by the IoT sensor and perform standardization processing; The real-time data points after the standardization process are encoded by the encoder of the encoding and decoding model to obtain a low-dimensional representation of the real-time data points; Reconstructing the low-dimensional representation of the real-time data point through the decoder of the encoding and decoding model to obtain a reconstructed data point; The difference between the real-time data point and the reconstructed data point is measured based on the Euclidean distance to obtain a reconstruction error, and a reconstruction standard deviation and a reconstruction mean are calculated based on the reconstruction error.

5. The method for processing Internet of Things data according to claim 4, characterized in that: The method of using a sliding window update mechanism to fill the reconstruction error into a preset sliding window of a predefined time length in chronological order includes: Organize the reconstruction errors by sorting by timestamps to obtain a list of reconstruction errors arranged in chronological order; Preliminarily filling the preset sliding window with the reconstruction error list arranged in chronological order to obtain a preset sliding window containing a preset number of reconstruction errors; The reconstruction error in the preset sliding window after preliminary filling is dynamically updated based on the sliding window update mechanism to keep the reconstruction error in the preset sliding window up to date.

6. The method for processing Internet of Things data according to claim 5, characterized in that: The values ​​of the first weight and the second weight are 0.3 and 0.7 respectively.

7. The method for processing Internet of Things data according to claim 6, characterized in that: The step of screening the reconstruction error corresponding to each real-time data point based on the dynamic threshold to obtain abnormal data points and cleaning the abnormal data points includes: The dynamic threshold is compared with the reconstruction error of the real-time data point. If the reconstruction error exceeds the dynamic threshold, the real-time data point corresponding to the reconstruction error is identified as an abnormal data point, and the abnormal data point is removed.

Images