Multi-time-series monitoring data anomaly detection method and system

Abstract

The invention relates to the field of data processing, in particular to a multi-time-series monitoring data anomaly detection method and system. A multi-time-series monitoring data anomaly detection system comprises a monitoring data set acquisition module, a monitoring data value change section division module, a trust data set establishment module, a monitoring data matching model establishment module and an abnormal value detection module. According to the method, the plurality of sensors are arranged, the monitoring data with the consistent change trend in the different monitoring data sets of the multi-time series are regarded as normal data, and the matching model is established based on the normal data of the fragments to identify the abnormal values in the monitoring data sets, so that the characteristics of the multi-time series data can be more accurately combined; and the identification accuracy is high.

Description

Technical Field

[0001] This invention relates to the field of data processing, and more specifically, to a method and system for detecting anomalies in multi-time series monitoring data. Background Technology

[0002] In today's interconnected world, people increasingly rely on analyzing monitoring data to observe changes in specific regions. For example, they analyze atmospheric monitoring data to observe pollution levels and water level data to observe flood season conditions. This monitoring data is primarily acquired from sensors. However, during the data acquisition process, the data uploaded by the sensors may differ significantly from the actual values ​​due to network transmission issues, which can severely impact data analysis. Summary of the Invention

[0003] This invention provides an abnormal traffic monitoring method and system based on convolutional networks. By deploying multiple sensors and treating monitoring data with consistent trends across different time series as normal data, a matching model is built based on these normal data segments to identify outliers in the monitoring dataset. This approach can more accurately combine the characteristics of multi-time series data and achieve high identification accuracy.

[0004] A method for anomaly detection in multi-time series monitoring data, comprising:

[0005] Obtain monitoring dataset X i This refers to data acquired by the same sensor deployed in the same area, where i represents the number of monitoring datasets, i∈{1, 2, 3...I}, and I represents the maximum number of monitoring datasets; monitoring dataset X i ={x i，1 x i，2 x i，3 ···x i，j ···x i，J}, where x i，j To monitor dataset X i The j-th monitoring data point in the dataset X, where j∈{1, 2, 3...J}, and J is the monitoring dataset X. i The total number of monitoring data in China, and x i，j =(t j V i，j ), where t j For monitoring data x i，j The corresponding time point, V i，j For monitoring data x i，j The corresponding monitoring data values;

[0006] Select monitoring dataset X one by one i For each monitoring dataset Xi Perform the following operation: Iterate through the selected monitoring dataset X i Monitoring data x i，j And according to monitoring data x i，j Extreme points and time points t in j Monitor dataset X i Divided into segments of changes in monitoring data values;

[0007] Based on the pairwise monitoring dataset X i The monitoring dataset X is determined by the range of changes in the monitoring data values ​​between them. i The corresponding trust dataset;

[0008] Select trust datasets Q one by one i and with the trust dataset Q i Used as a training set to train a monitoring data matching model;

[0009] The monitoring dataset X is sequentially analyzed using a monitoring data matching model. i Anomaly detection is performed on the monitoring data within the system.

[0010] As a preferred aspect of the present invention, based on monitoring data x i，j Extreme points and time points t in j Dividing the monitoring dataset into segments of varying monitoring data values ​​involves the following steps: traversing the monitoring dataset X i Monitoring data x i，j Corresponding monitoring data value V i，j And based on the monitoring data value V i，j Obtain the extreme point x i，k Where k∈{1, 2, 3...J}, extreme points refer to monitoring data points that are all greater than or all less than their adjacent monitoring data points. The time points t corresponding to all extreme points are then considered. k Arranged in ascending order, each adjacent time point t k The interval between these intervals constitutes the range of changes in the monitored data values, T. g g is used to record the number of segments of change in the monitored data values, g∈{1,2,3...G}, where G is the monitored dataset X. i The total number of corresponding monitoring data value change segments; monitoring data value change segment T g The form is (t) min , t max ), where t min To monitor the change in data value segment T g The minimum time point, t max To monitor the change in data value segment T g The largest point in time.

[0011] As a preferred aspect of the present invention, based on pairwise monitoring datasets X i The monitoring dataset X is determined by the range of changes in the monitoring data values ​​between them. i The corresponding trust dataset includes the following steps:

[0012] S1: Let r = 1, r is used as the number to select the monitoring dataset;

[0013] S2: Select monitoring dataset X r ;

[0014] S3: Let u = 1, u is used as a number to select the range of changes in the monitored data value;

[0015] S4: Select monitoring dataset X r The change segment T of the u-th monitoring data value in u The monitoring data value change segment T u Match the data with all the ranges of changes in the monitoring data values ​​in the remaining monitoring dataset, and calculate the number of successful matches, y.

[0016] S5: Determine whether "y≥Y" is true, where Y is the confidence threshold. If "y≥Y" is true, change the range T of this monitored data value. u Store the monitored data value change segment in the temporary storage set and proceed to S6; if "y≥Y" is not true, proceed directly to S6;

[0017] S6: Determine if "u < G" is true. If "u < G" is true, assign u+1 to u and return to S4; if "u < G" is false, proceed to S7.

[0018] S7: Select monitoring data value change segments one by one from the temporary set of monitoring data value change segments, and store all monitoring data corresponding to the time points within the selected monitoring data value change segments into the trusted dataset Q. r ;

[0019] S8: Determine if "r < I" is true. If "r < I" is true, assign r + 1 to r and return to S2; if "r < I" is false, proceed to S9.

[0020] S9: Output the complete trust dataset Q i .

[0021] As a preferred aspect of the present invention, the monitoring dataset X is sequentially processed using a monitoring data matching model. i Anomaly detection is performed on the monitoring data within the dataset, specifically including the following steps: Select each monitoring dataset X... i To monitor dataset X iThe first monitoring data is used as the initial value. The initial value is input into the monitoring data matching model, and the predicted values ​​are output sequentially, denoted as P1. The corresponding true value is denoted as P2. If "|P1-P2| / P2<μ" is true, u is the fluctuation threshold, and this true value is not an outlier. If "|P1-P2| / P2<μ" is not true, this true value is an outlier.

[0022] As a preferred aspect of the present invention, when generating the monitoring data value change segment T g At that time, the trend value B is used to analyze the change segment T of the monitored data value. g Label the data, B∈{0,1}, to represent the segment T of change in the monitored data value. g The trend of changes in the monitored data values, if the time point is t min and t max If the monitoring data values ​​change in an increasing trend, then B = 1; if the time point is at t... min and t max If the trend of the monitoring data values ​​between them is decreasing, then B = 0;

[0023] In step S4, when the monitored data value changes by segment T u When matching with all monitoring data value change segments in the other monitoring dataset, if the match is successful, it is determined whether the B values ​​corresponding to the two successfully matched monitoring data value change segments are consistent. If they are consistent, no operation is performed; if they are inconsistent, the match is considered a failure, and the monitoring data within the time points corresponding to the two successfully matched monitoring data value change segments are considered outliers.

[0024] As a preferred aspect of the present invention, the monitoring data matching model is established based on the RNN model.

[0025] A multi-time-series monitoring data anomaly detection system, comprising:

[0026] The monitoring dataset acquisition module is used to acquire monitoring datasets;

[0027] The monitoring data value change segment division module is used to divide the monitoring dataset into monitoring data value change segments based on the extreme points and time points in the monitoring data.

[0028] The Trust Dataset Establishment Module is used to determine the Trust Dataset corresponding to a Monitoring Dataset based on the range of changes in monitoring data values ​​between two monitoring datasets.

[0029] The monitoring data matching model building module is used to build a monitoring data matching model based on the trust dataset corresponding to the monitoring dataset.

[0030] The outlier detection module is used to sequentially detect outliers in the monitoring data within the monitoring dataset using a monitoring data matching model.

[0031] The present invention has the following advantages:

[0032] 1. This invention deploys multiple sensors and treats monitoring data with consistent trends across different time series as normal data. Based on these normal data segments, a matching model is built to identify outliers in the monitoring dataset. This approach can more accurately combine the characteristics of multi-time series data and achieve high identification accuracy.

[0033] 2. This invention matches the changing trends of monitoring data values ​​and treats monitoring data with inconsistent changing trends as outliers, thus avoiding data errors caused by sensor malfunctions from affecting subsequent data analysis. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the structure of the multi-time series monitoring data anomaly detection system used in an embodiment of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of this invention.

[0036] Example 1

[0037] A method for anomaly detection in multi-time series monitoring data, comprising:

[0038] Obtain monitoring dataset X i This refers to data acquired by the same sensor deployed in the same area, where i represents the number of monitoring datasets, i∈{1, 2, 3...I}, and I represents the maximum number of monitoring datasets; monitoring dataset X i ={x i，1 x i，2 x i，3 ···x i，j ···x i，J}, where x i，j To monitor dataset X i The j-th monitoring data point in the dataset X, where j∈{1, 2, 3...J}, and J is the monitoring dataset X. i The total number of monitoring data in China, and x i，j =(t j V i，j ), where t j For monitoring data x i，j The corresponding time point, V i，j For monitoring data x i，j The corresponding monitoring data values;

[0039] Select monitoring dataset X one by one i For each monitoring dataset X i Perform the following operation: Iterate through the selected monitoring dataset X i Monitoring data x i，j And according to monitoring data x i，j Extreme points and time points t in j Monitor dataset X i Divided into segments of changes in monitoring data values;

[0040] According to monitoring data x i，j Extreme points and time points t in j Dividing the monitoring dataset into segments of varying monitoring data values ​​involves the following steps: traversing the monitoring dataset X i Monitoring data x i，j Corresponding monitoring data value V i，j And based on the monitoring data value V i，j Obtain the extreme point x i，k Where k∈{1, 2, 3...J}, extreme points refer to monitoring data points that are all greater than or all less than the adjacent monitoring data points. For example, in a series of monitoring data points {0.53, 0.67, 0.69, 0.64, 0.55}, the extreme points are the monitoring data points corresponding to 0.53, 0.69, and 0.55. The time points t corresponding to all extreme points are then considered. k Arranged in ascending order, each adjacent time point t k The interval between these intervals constitutes the range of changes in the monitored data values, T. g g is used to record the number of segments of change in the monitored data values, g∈{1,2,3...G}, where G is the monitored dataset X. i The total number of corresponding monitoring data value change segments; for example, the time points corresponding to the extreme points are t2, t5, t8, and t... 11 The corresponding monitoring data change segments are (t2, t5), (t5, t8), and (t8, t5), respectively. 11 ); Monitoring data value change segment T g The form is (t) min , t max ), where t min To monitor the change in data value segment T g The minimum time point, t max To monitor the change in data value segment T g The largest point in time;

[0041] Based on the pairwise monitoring dataset X i The monitoring dataset X is determined by the range of changes in the monitoring data values ​​between them. i The corresponding trust dataset;

[0042] Based on the pairwise monitoring dataset X i The monitoring dataset X is determined by the range of changes in the monitoring data values ​​between them. i The corresponding trust dataset includes the following steps:

[0043] S1: Let r = 1, r is used as the number to select the monitoring dataset;

[0044] S2: Select monitoring dataset X r ;

[0045] S3: Let u = 1, u is used as a number to select the range of changes in the monitored data value;

[0046] S4: Select monitoring dataset X r The change segment T of the u-th monitoring data value in u The monitoring data value change segment T u Match the data with all the ranges of changes in the monitoring data values ​​in the remaining monitoring dataset, and calculate the number of successful matches, y.

[0047] S5: Determine whether "y≥Y" is true, where Y is the confidence threshold, set by the operator, typically (0.7*I). If "y≥Y" is true, it indicates that the change range T of this monitored data value is within the range... u The changes in the internal monitoring data values ​​have a high confidence level, and this range of changes in the monitoring data values, T... u Store the monitored data value change segment in the temporary storage set and proceed to S6; if "y≥Y" is not true, it indicates that this monitored data value change segment T... u Changes in internal monitoring data values ​​do not have a high degree of confidence and proceed directly to S6;

[0048] S6: Determine if "u < G" is true. If "u < G" is true, it means that the monitoring dataset X is true. r If the monitored data value change segment in the dataset has not been completely traversed, assign u+1 to u and return to S4; if "u < G" is not true, it means that the monitored dataset X... r The monitoring data value change segments have been completely traversed; proceed to S7.

[0049] S7: Select monitoring data value change segments one by one from the temporary set of monitoring data value change segments, and store all monitoring data corresponding to the time points within the selected monitoring data value change segments into the trusted dataset Q. r Trust dataset Q r The fact that the monitoring data within the system can be verified in other sensors indicates that the probability of these monitoring data being affected by factors such as network transmission failure or sensor failure is relatively small, and that they have higher data value.

[0050] S8: Determine if "r < I" is true. If "r < I" is true, it means that not all monitoring datasets have been traversed. Assign r + 1 to r and return to S2. If "r < I" is false, it means that all monitoring datasets have been traversed. Proceed to S9.

[0051] S9: Output the complete trust dataset Q i ;

[0052] Select trust datasets Q one by one i and with the trust dataset Q i The monitoring data matching model is trained using the training set and is based on an RNN model.

[0053] The monitoring dataset X is sequentially analyzed using a monitoring data matching model. i The monitoring data within the system is used for outlier detection; it should be noted that this outlier refers to monitoring data values ​​that differ significantly from the true values ​​due to network transmission issues or sensor malfunctions.

[0054] The monitoring dataset X is sequentially analyzed using a monitoring data matching model. i Anomaly detection is performed on the monitoring data within the dataset, specifically including the following steps: Select each monitoring dataset X... i To monitor dataset X i The first monitoring data is used as the initial value. This is because when the sensor is first used, the data is considered reliable due to on-site debugging by operators. The initial value is input into the monitoring data matching model, and the predicted values ​​are output sequentially, denoted as P1, and the corresponding actual values ​​are denoted as P2. If "|P1-P2| / P2<μ" is true, where u is the fluctuation threshold set by the operator, it means that the predicted value and the actual value are not significantly different, and this actual value is not an outlier. If "|P1-P2| / P2<μ" is not true, it means that the predicted value and the actual value differ significantly, and this actual value is an outlier.

[0055] In this application, each monitoring dataset is a time series dataset. When monitoring data in a certain area, the deployed sensors may be affected by network transmission failures, resulting in outliers in the monitoring dataset. This application deploys multiple sensors and treats monitoring data with consistent trends in different time series datasets as normal data. Based on these normal data segments, a matching model is established to identify outliers in the monitoring dataset. This approach can more accurately combine the characteristics of multi-time series data and achieve high identification accuracy.

[0056] When generating the monitoring data value change segment, T g At that time, the trend value B is used to analyze the change segment T of the monitored data value. g Label the data, B∈{0,1}, to represent the segment T of change in the monitored data value.g The trend of changes in the monitored data values, if the time point is t min and t max If the monitoring data values ​​change in an increasing trend, then B = 1; if the time point is at t... min and t max If the trend of the monitoring data values ​​between them is decreasing, then B = 0;

[0057] In step S4, when the monitored data value changes by segment T u When matching with all monitoring data value change segments in the other monitoring datasets, if the match is successful, it is determined whether the B values ​​corresponding to the two successfully matched monitoring data value change segments are consistent. If they are consistent, it means that the change trends of the two successfully matched monitoring data value change segments are consistent, and no operation is performed. If they are inconsistent, it means that the change trends of the two successfully matched monitoring data value change segments are inconsistent, and this match is considered a failure. Furthermore, the monitoring data within the time points corresponding to the two successfully matched monitoring data value change segments are considered outliers.

[0058] In this application, the changing trends of monitoring data values ​​are matched, and monitoring data with inconsistent changing trends are regarded as outliers, so as to avoid data errors caused by sensor failure affecting subsequent data analysis.

[0059] Example 2

[0060] A multi-time-series monitoring data anomaly detection system, such as Figure 1 As shown, it includes:

[0061] The monitoring dataset acquisition module is used to acquire monitoring datasets;

[0062] The monitoring data value change segment division module is used to divide the monitoring dataset into monitoring data value change segments based on the extreme points and time points in the monitoring data.

[0063] The Trust Dataset Establishment Module is used to determine the Trust Dataset corresponding to a Monitoring Dataset based on the range of changes in monitoring data values ​​between two monitoring datasets.

[0064] The monitoring data matching model building module is used to build a monitoring data matching model based on the trust dataset corresponding to the monitoring dataset.

[0065] The outlier detection module is used to sequentially detect outliers in the monitoring data within the monitoring dataset using a monitoring data matching model.

[0066] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims. Parts not described in detail in this specification are prior art known to those skilled in the art.

Claims

1. A method for detecting anomalies in monitoring data of multiple time series, characterized in that: include: Get monitoring dataset X i , is used to refer to the data acquired by the same sensor located in the same area, where i refers to the number of monitoring data sets, i∈{1, 2, 3······I}, and I refers to the maximum number of monitoring data sets; the monitoring data set X i ={x i，1 , x i，2 , x i，3 ···x i，j ···x i，J }, where x i，j is the monitoring dataset X i The jth monitoring data in X, j∈{1, 2, 3·····J}, J is the monitoring data set X i The total number of monitoring data in x i，j =(t j , V i，j ), where t j The monitoring data x i，j The corresponding time point, V i，j The monitoring data x i，j The corresponding monitoring data value; Select monitoring datasets X one by one i , for each monitoring data set X i Perform the following operations: Traverse the selected monitoring data set X i Monitoring data in x i，j , and according to the monitoring data x i，j The extreme point and time point t in j The monitoring data set X i Divided into monitoring data value change segments; According to the pairwise monitoring data set X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust dataset; Select trust datasets Q one by one i , and the trust dataset Q i Use it as a training set to train the monitoring data matching model; Through the monitoring data matching model, the monitoring data set X i The monitoring data in the system are used for outlier detection.

2. The method for detecting anomalies in monitoring data of multiple time series according to claim 1, characterized in that: According to the monitoring data x i，j The extreme point and time point t in j The monitoring data set is divided into monitoring data value change segments, which specifically include the following steps: traverse the monitoring data set X i Monitoring data x i，j The corresponding monitoring data value V i，j , and according to the monitoring data value V i，j Get the extreme point x i，k , where k∈{1, 2, 3······J}, extreme value points refer to the monitoring data corresponding to values ​​that are greater than or less than the adjacent monitoring data values. The time points t corresponding to all extreme value points are k Arrange them in order from small to large, and each adjacent time point t k The monitoring data value change segment T g , g is used to record the number of change segments of the monitoring data value, g∈{1, 2, 3······G}, G is the monitoring data set X i The total number of corresponding monitoring data value change segments; monitoring data value change segment T g The form is (t min , t max ), where t min To monitor the data value change period T g The smallest time point, t max To monitor the data value change period T g The largest time point.

3. The method for detecting anomalies in monitoring data of multiple time series according to claim 2, characterized in that: According to the pairwise monitoring data set X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust data set specifically includes the following steps: S1: Let r = 1, r is used as the number to select the monitoring data set; S2: Select monitoring dataset X r ; S3: Let u=1, where u is used as a number to select a monitoring data value change segment; S4: Select monitoring dataset X r The uth monitoring data value change segment T u , the monitoring data value change segment T u Match with all monitoring data value change segments in the rest of the monitoring data set, and calculate the number of successful matches y; S5: Determine whether "y≥Y" is true, where Y is the confidence threshold. If "y≥Y" is true, change the value of this monitoring data by segment T u Store the monitored data value change segment in the temporary storage set and enter S6; if "y≥Y" is not true, directly enter S6； S6: Determine whether "u<G" holds. If so, assign u+1 to u and return to S4. If not, go to S7. S7: Select monitoring data value change segments one by one from the monitoring data value change segment temporary storage set, and store all monitoring data corresponding to the time points within the selected monitoring data value change segments into the trust data set Q r ; S8: Determine whether "r<I" holds. If so, assign r+1 to r and return to S2; if not, enter S9; S9: Output the entire trust data set Q i .

4. The method for detecting anomalies in monitoring data of multiple time series according to claim 3, characterized in that: Through the monitoring data matching model, the monitoring data set X i The monitoring data in the system is used for outlier detection, which specifically includes the following steps: Select monitoring datasets X one by one i , to monitor the dataset X i The first monitoring data in is taken as the initial value, the initial value is input into the monitoring data matching model, and the predicted value is output in sequence, denoted as P1, and the corresponding true value is denoted as P2. If "|P1-P2| / P2<μ" holds, u is the fluctuation threshold, and this true value is not an abnormal value; if "|P1-P2| / P2<μ" does not hold, this true value is an abnormal value.

5. The method for detecting anomalies in monitoring data of multiple time series according to claim 4, characterized in that: When generating the monitoring data value change segment T g When the change trend B value is used to monitor the change segment T of the data value g Mark, B∈{0,1}, to characterize the monitoring data value change segment T g In the monitoring data value change trend, if the time point is t min and t max If the monitoring data value change trend between t and t is an increasing trend, then B = 1; if the time point is t min and t max If the monitoring data value change trend between is a decreasing trend, then B=0; In step S4, when the monitoring data value changes to segment T u When matching with all the monitoring data value change segments in the remaining monitoring data sets, if the match is successful, determine whether the B values ​​corresponding to the two successfully matched monitoring data value change segments are consistent. If they are consistent, no operation is performed; if they are inconsistent, the match is regarded as a matching failure, and the monitoring data within the corresponding time points of the two successfully matched monitoring data value change segments are regarded as abnormal values.

6. The method for detecting anomalies in monitoring data of multiple time series according to claim 5, characterized in that: The monitoring data matching model is established based on the RNN model.

7. A multi-time series monitoring data anomaly detection system, characterized in that: include: A monitoring data set acquisition module, used to acquire a monitoring data set; A monitoring data value change segment division module is used to divide the monitoring data set into monitoring data value change segments according to extreme value points and time points in the monitoring data; A trust data set establishment module is used to determine the trust data set corresponding to the monitoring data set according to the monitoring data value change segment between two monitoring data sets; A monitoring data matching model building module is used to build a monitoring data matching model according to a trust data set corresponding to the monitoring data set; The outlier detection module is used to perform outlier detection on the monitoring data in the monitoring data set in turn through the monitoring data matching model.

Images