Sensor data drift online calibration method

Abstract

A sensor data drift on-line calibration method comprises the steps of data preprocessing, hierarchical clustering analysis, correlation analysis and drift preliminary identification, time sequence prediction based on Transform and on-line calibration, and the accuracy of sensor data of the Internet of Things can be effectively improved. A time sequence data correlation analysis and clustering algorithm is utilized, a sensor with data drifting is intelligently identified, a time sequence prediction algorithm based on Transform is adopted, time and space correlation of sensor data is accurately captured, and high-precision prediction is achieved. And through an online calibration mechanism, the data drift is responded in real time, and the real-time accuracy of the data is ensured. In addition, the calibrated data are re-integrated into the data set, and the prediction model is updated, so that the adaptive updating of the model is realized, and the adaptability and accuracy of the system are further improved. The method is of great significance in improving the reliability of sensor data and prolonging the service life of the sensor network, and is particularly suitable for the application of the Internet of Things needing long-term stable operation.

Description

Technical Field

[0001] The invention relates to Internet of Things data acquisition and sensor drift calibration technology, and in particular to a sensor data drift online calibration method. Background Art

[0002] The accuracy of IoT data collection is a prerequisite for subsequent analysis and application. However, since the sensor measurement nodes mostly use relatively cheap and easy-to-deploy sensors, and are often deployed in relatively harsh environments such as the wild and inside buildings, it is easy to produce measurement errors and data drift. With the deployment of new IoT infrastructure, higher requirements are placed on the operating time, real-time monitoring capabilities, and data accuracy of sensor networks. However, as the operating time increases, the drift and error accumulation problems generated by sensors will become more serious, greatly affecting the credibility of IoT data.

[0003] Traditional field calibration methods require sensors to be uninstalled from the system or to be manually carried to the site with special calibration instruments, which results in high calibration costs and affects the real-time nature of data. Current research on online calibration algorithms focuses on dense networks, mobile networks, and target detection networks. These three types of networks provide references for blind calibration through their special application properties, so their application scope is limited. In addition, the existing algorithms do not control prediction errors, so the accuracy of their calibration results is difficult to guarantee. In some experiments, this method can even make the calibration error greater than the drift error.

[0004] Therefore, developing an online calibration method for sensor data drift based on time series prediction is of great significance for improving the sensor calibration accuracy and realizing a universal online calibration method for IoT data.

[0005] It should be noted that the information disclosed in the above background technology section is only used for understanding the background of the present application, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the invention

[0006] The main purpose of the present invention is to overcome the defects existing in the above-mentioned background technology and provide an online calibration method for sensor data drift.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] A sensor data drift online calibration method comprises the following steps:

[0009] S1. Preprocess the collected sensor data;

[0010] S2. Use hierarchical clustering method to group the historical data before time T;

[0011] S3. For the historical data before time T, calculate the correlation matrix of the sensor data in the same group, regularly check the matrix changes, and set the threshold for preliminary drift identification and screening to identify potential drift sensors;

[0012] S4. Using the Transformer-based time series prediction algorithm, using the correlation matrix, capturing the temporal correlation and spatial correlation of the sensor data, and generating a high-precision prediction value after time T based on the historical data before time T and its grouping results;

[0013] S5. Perform correlation analysis and cluster analysis on the measured data after time T again, group the sensor time series data, and when the grouping result changes compared with the grouping result before time T, and the difference between the predicted value after time T and the measured data after time T exceeds the preset threshold, trigger the online calibration mechanism and use the predicted value instead of the measured value for calibration.

[0014] Furthermore, the method further comprises the following steps:

[0015] S6. Add the calibrated data to the overall data set, and retrain the time series prediction model based on the new data set to achieve adaptive updating of the model.

[0016] Furthermore, in step S1, the initial reliable data is processed using wavelet transform to remove noise and extract features.

[0017] Furthermore, in step S1, the collected sensor data is preprocessed using LSTM and interpolation filling method.

[0018] Furthermore, in step S2, data similarity is measured by dynamic time warping (DTW), and average link is used as link function to identify data patterns and achieve grouping.

[0019] Furthermore, in step S4, the Transformer-based time series prediction algorithm specifically includes: utilizing a seasonal decomposition method based on local weighted regression to decompose the time series into three parts: trend, season, and residual, and updating the LOESS smoothing parameter through multiple iterations to gradually improve the decomposition accuracy; and adding a frequency enhancement module to apply the Transformer's attention mechanism to the frequency domain to capture the global properties of the time series.

[0020] Furthermore, in step S4, the Transformer architecture of the time series prediction algorithm only includes the encoder part, which captures the unique change trend between variables through a variable-based feature embedding method; applies layer normalization operations to enhance the model's adaptability to non-aligned data; uses a feedforward network to replace time series dependency modeling to extract the periodicity and amplitude characteristics of the time series; and introduces a self-attention module to improve the modeling effect of multi-dimensional time series data.

[0021] Furthermore, step S4 also includes reconstructing the Transformer structure by combining the time series signal season-trend decomposition and frequency conversion module to verify the effectiveness of time series prediction.

[0022] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the sensor data drift online calibration method is implemented.

[0023] A computer program product comprises a computer program, wherein when the computer program is executed by a processor, the method for online calibration of sensor data drift is implemented.

[0024] The present invention has the following beneficial effects:

[0025] The present invention proposes an online calibration method for sensor data drift, which can effectively improve the accuracy of IoT sensor data and meet the high requirements of new infrastructure for sensor network operation time, real-time monitoring capabilities and data accuracy. The method uses time series data correlation analysis and clustering algorithms to intelligently identify drifting sensors, and uses a Transformer-based time series prediction algorithm to accurately capture the time correlation and spatial correlation of sensor data, thereby improving prediction accuracy. Compared with traditional field calibration methods, the online calibration algorithm of the present invention avoids the high calibration cost and the negative impact on the real-time performance of data, while solving the problems of limited application scope of existing online calibration algorithms and difficulty in ensuring the correctness of calibration results. In addition, by adding the calibrated data to the overall data set and retraining the time series prediction model, the adaptive update of the model is achieved, which further improves the reliability of sensor data and effectively extends the service life of the sensor network. The successful implementation of this method provides a universal solution for the online calibration of IoT data, which is of great significance for improving the credibility of IoT data.

[0026] Other beneficial effects of the embodiments of the present invention will be further described below. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Schematic diagram of sensor data drift.

[0028] Figure 24 is a framework diagram of an online calibration algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0030] An embodiment of the present invention provides a sensor data drift online calibration method, comprising the following steps:

[0031] Step S1. Data preprocessing: preprocess the collected sensor data.

[0032] In a preferred embodiment, in step S1, the initial reliable data is processed using wavelet transform to remove noise and extract features. Further, the collected sensor data is preprocessed using LSTM and interpolation filling method to fill in missing data.

[0033] Step S2. Hierarchical clustering analysis: Use the hierarchical clustering method to group the historical data before time T.

[0034] In a preferred embodiment, in step S2, data similarity is measured by dynamic time warping (DTW), and average link is used as the link function to identify data patterns and achieve grouping.

[0035] Step S3. Correlation analysis and initial drift identification: For historical data before time T, calculate the correlation matrix of sensor data in the same group, regularly check the matrix changes, and set thresholds for initial drift identification screening to identify potential drift sensors;

[0036] Step S4. Transformer-based time series prediction: using the Transformer-based time series prediction algorithm, using the correlation matrix, capturing the time correlation and spatial correlation of the sensor data, and generating a high-precision prediction value after time T based on the historical data before time T and its grouping results;

[0037] In a preferred embodiment, in step S4, the Transformer-based time series prediction algorithm specifically includes: utilizing a seasonal decomposition method based on local weighted regression to decompose the time series into three parts: trend, season, and residual, and updating the LOESS smoothing parameter through multiple iterations to gradually improve the decomposition accuracy; and adding a frequency enhancement module to apply the Transformer's attention mechanism to the frequency domain to capture the global properties of the time series.

[0038] In a preferred embodiment, the Transformer architecture of the time series prediction algorithm only includes the encoder part, which captures the unique change trends between variables through a variable-based feature embedding method; applies layer normalization operations to enhance the model's adaptability to non-aligned data; uses a feedforward network to replace time series dependency modeling to extract the periodicity and amplitude characteristics of the time series; and introduces a self-attention module to improve the modeling effect of multi-dimensional time series data.

[0039] In a preferred embodiment, the seasonal-trend decomposition and frequency conversion modules of time series signals are combined to reconstruct the Transformer structure and verify the effectiveness of time series prediction.

[0040] Step S5. Online calibration: perform correlation analysis and cluster analysis on the measured data after time T again, group the sensor time series data, and when the grouping result changes compared with the grouping result before time T, and the difference between the predicted value after time T and the measured data after time T exceeds the preset threshold, trigger the online calibration mechanism and use the predicted value instead of the measured value for calibration.

[0041] Step S6. Adaptive update of network parameters: Add the calibrated data to the overall data set, and retrain the time series prediction model based on the new data set to achieve adaptive update of the model.

[0042] This paper proposes an innovative online calibration method for sensor data drift, which effectively solves the problem of sensor data accuracy in the Internet of Things environment. Through a series of carefully designed steps, this method realizes real-time monitoring and calibration of sensor data, thereby improving the reliability of data and the service life of the sensor network. By using time series data correlation analysis and clustering algorithms, sensors with data drift are intelligently identified, and a Transformer-based time series prediction algorithm is used to accurately capture the temporal and spatial correlation of sensor data. This method not only has good prediction accuracy, but also can respond to data drift in real time through an online calibration mechanism to ensure the real-time accuracy of data.

[0043] The method can effectively deal with the drift problem of sensors in harsh environments, avoiding the high cost of traditional calibration methods and the negative impact on data real-time performance. Compared with the existing online calibration algorithm, the method of the present invention has a wider applicability, is not limited to a specific type of network, and ensures the correctness of the calibration result through precise prediction error control. In addition, the method realizes the adaptive update of the model by reintegrating the calibrated data into the data set and updating the prediction model, further improving the adaptability and accuracy of the system. The online calibration method for sensor data drift of the present invention is of great significance for Internet of Things applications that require long-term stable operation. It can significantly improve the reliability of sensor data and extend the service life of the sensor network, which is particularly critical for key application areas such as bridge structure health detection and environmental monitoring. By deploying the algorithm to the cloud, online drift identification and regular calibration can be achieved, further improving the performance and stability of the Internet of Things system, and providing a strong guarantee for the accuracy and real-time performance of Internet of Things data.

[0044] The specific embodiments and experimental verification of the present invention are further described below.

[0045] The present invention designs an online calibration algorithm for sensor network data drift. It uses time series data correlation analysis and clustering algorithm to identify drift sensors, and uses a Transformer-based time series prediction algorithm to capture the time correlation of sensor time series data and the spatial correlation between different sensors, improve the prediction accuracy, and calibrate the sensor data that triggers the online calibration mechanism.

[0046] The idea of ​​the method proposed in the present invention is to select a period of historical data before time T for correlation analysis and cluster analysis, and divide the N-dimensional sensor time series data into M1 groups, and use the Transformer-based time series prediction algorithm to obtain the predicted value after time T, and perform correlation analysis and cluster analysis on the measured data after time T again, and divide the N-dimensional sensor time series data into M2 groups. When the grouping changes and the difference between the predicted value and the measured value exceeds a certain threshold, the online calibration mechanism of the sensor data is triggered, and the predicted value is used instead of the measured value for calibration. Retrain the network to adaptively update the network parameters.

[0047] Clustering the initial data: Hierarchical clustering is a clustering method that gradually merges or splits the hierarchical structure of data. It can be used to group and cluster time series data without knowing the number of groups in advance. In hierarchical clustering, the similarity measure and the choice of link function have a great influence on the final clustering results. Considering the characteristics of time series data, dynamic time warping (DTW) is selected to measure similarity and average link is used as the link function.

[0048] Calculate the correlation of sensor data in the same group: calculate the symbiosis coefficient matrix between the same group, regularly check the changes in the symbiosis coefficient matrix, and set thresholds for preliminary identification and screening of drift.

[0049] Establish a Transformer-based time series prediction algorithm:

[0050] 1. Time series decomposition. The seasonal decomposition method based on local weighted regression is used to decompose the time series into three parts: trend, season and residual. Multiple iterations are performed, and the parameters of LOESS smoothing are updated in each iteration to gradually improve the accuracy of decomposition.

[0051] 2. Frequency transformation. To further improve the long-term prediction performance of Transformer, we take advantage of the fact that most time series predictions have sparse representation after frequency transformation, add a frequency enhancement module to improve the algorithm architecture, and apply the attention mechanism of Transformer to the frequency domain, which helps to better capture the global properties of the time series. In addition to making the algorithm more suitable for long-term prediction, combining Transformer with frequency analysis also reduces the complexity of the algorithm from quadratic to linear.

[0052] 3. Architecture adjustment. More efficient time series modeling is achieved through structural adjustment. Different from the traditional encoder-decoder structure, the adjusted architecture only contains the encoder part, which simplifies the model and improves computational efficiency. The variable-based feature embedding method is adopted to make each variable independently represented, which helps to capture the unique change trend between variables. The layer normalization operation is dedicated to the variable feature vector to avoid noise interference between different time steps and enhance the adaptability of the model to non-aligned data. The feedforward network replaces the time series dependency modeling, and extracts the common features of the time series such as periodicity and amplitude through its powerful expression ability, achieving good generalization of unseen data. At the same time, the self-attention module improves the modeling effect of multi-dimensional time series data through the feature interaction between variables, so that the model performs well in long time series prediction and reduces the computational complexity.

[0053] Combining the seasonal-trend decomposition and frequency transformation modules of time series signals, the Transformer structure is reconstructed, and the effectiveness of the above three modules for time series prediction is verified respectively. The improved algorithm can improve the model's modeling ability for time series data and better capture the long-term dependencies and dynamic changes in the sequence. More accurate predictions can help to better calibrate IoT data.

[0054] like Figure 2 As shown, the online calibration algorithm mainly includes the following processes:

[0055] 1. Initial reliable data collection: Data is collected from the sensor network over a period of time, which is considered to be reliable data without drift.

[0056] 2. Wavelet transform processing: Perform wavelet transform processing on the collected initial reliable data to remove noise and extract features to improve the quality of the data.

[0057] 3. Correlation modeling analysis: Use correlation modeling to analyze data and identify dependencies and potential drift patterns between data.

[0058] 4. Data processing to be verified: For new sensor data, verification is required to calculate the error between the actual data and the predicted data.

[0059] 5. Error accumulation calculation: Calculate the cumulative error between actual data and predicted data within a period of time t

[0060] 6. Threshold comparison judgment: The calculated error accumulation value After standardization Compare it with the preset threshold Q to determine whether calibration is needed.

[0061] 7. Time series prediction execution: If the accumulated error value is less than the threshold, continue to use the current prediction model for time series prediction.

[0062] 8. Online calibration mechanism trigger: If the error accumulation value is greater than or equal to the threshold, the online calibration mechanism is triggered and the predicted value is used. Replace the measured value Perform calibration.

[0063] 9. Network parameter update: After calibration, the calibration data is added to the overall data set, and the time series prediction model is retrained based on the new data set to achieve adaptive update of the model, improve the reliability of sensor data and extend the service life of the sensor network.

[0064] 10. Waiting for the next test: After the calibration and parameter update are completed, the algorithm enters a waiting state until the next data collection cycle begins and repeats the above process.

[0065] Experimental example

[0066] Sensor data from the initial layout of the Hutuo River Bridge Structure Health Detection System for one year was collected. Data was sampled every 10 minutes, with a total of 113 sensor data and more than 50,000 data points accumulated to form a complete data set. The data set was preprocessed using LSTM and interpolation filling methods. In addition, a simulated data set was created. 20 sensors and 5 signal sources were randomly arranged in a circular area with a radius of 1. The drift-free data received by the sensor was generated by the weighted addition of all signal sources in the area. The signal source was generated by an ARMA low-pass filter, and a random signal was added to each signal source to simulate the differences between signal sources in the real environment. Some sensors were artificially selected to superimpose drift signals as real measurement values, with different superposition times. In this way, a simulated sensor data set with 20 dimensions and 10,000 data points was obtained to verify the reliability of the drift identification method.

[0067] For the Hutuo River bridge monitoring data set, the initial 4 months of data are considered to be reliable data without drift. Based on this segment of data, initial hierarchical clustering analysis and correlation matrix calculation are performed, and the initial time series prediction model is trained. Drift detection is subsequently performed every 10 days. If a drifting sensor is identified, the data is deleted, and predictions are made based on past sensor network data and other sensor data in this time period, as well as the trained time series prediction model, to complete the calibration. The calibrated data is added to the overall data set and the time series prediction model is retrained to achieve adaptive updates of the model. The algorithm is deployed to the cloud of the bridge structure health detection system for online drift identification and regular calibration to improve the reliability of sensor data and extend the service life of the sensor network.

[0068] In summary, the present invention provides an online calibration method for sensor data drift, which can effectively improve the accuracy of IoT sensor data. By using time series data correlation analysis and clustering algorithms, sensors with data drift are intelligently identified, and a Transformer-based time series prediction algorithm is used to accurately capture the temporal and spatial correlation of sensor data to achieve high-precision prediction. Through the online calibration mechanism, data drift is responded to in real time to ensure the real-time accuracy of the data. In addition, the calibrated data is reintegrated into the data set and the prediction model is updated to achieve adaptive update of the model, further improving the adaptability and accuracy of the system. The present invention is of great significance for improving the reliability of sensor data and extending the service life of sensor networks, and is particularly suitable for IoT applications that require long-term stable operation.

[0069] An embodiment of the present invention further provides a storage medium for storing a computer program, which at least performs the above method when executed.

[0070] An embodiment of the present invention further provides a control device, comprising a processor and a storage medium for storing a computer program; wherein the processor is configured to execute at least the method described above when executing the computer program.

[0071] An embodiment of the present invention further provides a processor, wherein the processor executes a computer program and at least executes the method described above.

[0072] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a ferromagnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The storage medium described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0073] In the several embodiments provided by the present invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0074] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0075] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0076] Those skilled in the art can understand that: all or part of the steps of implementing the above method embodiments can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiments; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), disks or optical disks, etc. Various media that can store program codes.

[0077] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0078] The methods disclosed in the several method embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0079] The features disclosed in several product embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0080] The features disclosed in several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0081] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, several equivalent substitutions or obvious variations can be made without departing from the concept of the present invention, and the performance or use is the same, which should be regarded as belonging to the protection scope of the present invention.

Claims

1. A sensor data drift online calibration method, characterized in that: The following steps are involved: S1. Preprocess the collected sensor data; S2. Use hierarchical clustering method to group the historical data before time T; S3. For the historical data before time T, calculate the correlation matrix of the sensor data in the same group, regularly check the matrix changes, and set the threshold for preliminary drift identification and screening to identify potential drift sensors; S4. Using the Transformer-based time series prediction algorithm, using the correlation matrix, capturing the temporal correlation and spatial correlation of the sensor data, and generating a high-precision prediction value after time T based on the historical data before time T and its grouping results; S5. Perform correlation analysis and cluster analysis on the measured data after time T again, group the sensor time series data, and when the grouping result changes compared with the grouping result before time T, and the difference between the predicted value after time T and the measured data after time T exceeds the preset threshold, trigger the online calibration mechanism and use the predicted value instead of the measured value for calibration.

2. The sensor data drift online calibration method according to claim 1, characterized in that: The following steps are also included: S6. Add the calibrated data to the overall data set, and retrain the time series prediction model based on the new data set to achieve adaptive updating of the model.

3. The sensor data drift online calibration method according to claim 1 or 2, characterized in that: In step S1, the initial reliable data is processed using wavelet transform to remove noise and extract features.

4. The sensor data drift online calibration method according to any one of claims 1 to 3, characterized in that: In step S1, the collected sensor data is preprocessed using LSTM and interpolation filling method.

5. The sensor data drift online calibration method according to any one of claims 1 to 4, characterized in that: In step S2, data similarity is measured by dynamic time warping (DTW), and average link is used as the link function to identify data patterns and achieve grouping.

6. The sensor data drift online calibration method according to any one of claims 1 to 5, characterized in that: In step S4, the Transformer-based time series prediction algorithm specifically includes: using a seasonal decomposition method based on local weighted regression to decompose the time series into three parts: trend, season and residual, and updating the LOESS smoothing parameter through multiple iterations to gradually improve the decomposition accuracy; and adding a frequency enhancement module to apply the Transformer's attention mechanism to the frequency domain to capture the global properties of the time series.

7. The sensor data drift online calibration method according to claim 6, characterized in that: In step S4, the Transformer architecture of the time series prediction algorithm only includes the encoder part, which captures the unique change trend between variables through the variable-based feature embedding method; applies layer normalization operation to enhance the adaptability of the model to non-aligned data; uses a feedforward network to replace time series dependency modeling to extract the periodicity and amplitude characteristics of the time series; introduces a self-attention module to improve the modeling effect of multi-dimensional time series data.

8. The sensor data drift online calibration method according to claim 6 or 7, characterized in that: Step S4 also includes reconstructing the Transformer structure by combining the time series signal season-trend decomposition and frequency conversion module to verify the effectiveness of time series prediction.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for online calibration of sensor data drift according to any one of claims 1 to 8 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for online calibration of sensor data drift according to any one of claims 1 to 8 is implemented.

Images