A multi-time sequence monitoring data anomaly detection method and system

Abstract

The present application relates to the field of data processing, more particularly, it relates to a kind of multi-time series monitoring data anomaly detection method and system.A kind of multi-time series monitoring data anomaly detection system, comprising: monitoring data set acquisition module, monitoring data value change section division module, trust data set establishment module, monitoring data matching model establishment module and abnormal value detection module.The present application is by laying multiple sensors, and the change trend consistent monitoring data in different monitoring data set of multi-time series is regarded as normal data, and based on these segments normal data to establish matching model to identify abnormal value in monitoring data set, can be more accurately combined with the characteristics of multi-time series data, and identification accuracy is high.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more particularly, to a method and system for detecting anomalies in multi-time series monitoring data. Background Art

[0002] With the development of the internet, people are increasingly inclined to analyze monitoring data to observe changes in their respective areas. For example, they analyze atmospheric monitoring data to observe pollution levels and water level data to observe flood season conditions. These monitoring data are generally obtained from sensors. During the process of obtaining monitoring data through sensors, the data uploaded by the sensors may differ significantly from the actual value due to network transmission and other factors. This can have a significant impact on data analysis. Summary of the Invention

[0003] The present invention provides a convolutional network-based abnormal traffic monitoring method and system. By deploying multiple sensors, monitoring data with consistent change trends in different monitoring data sets of multiple time series are regarded as normal data, and a matching model is established based on these fragments of normal data to identify abnormal values ​​in the monitoring data set. This method can more accurately combine the characteristics of multiple time series data and has a high recognition accuracy.

[0004] A method for detecting anomalies in monitoring data of multiple time series, comprising:

[0005] Get monitoring dataset X i , is used to refer to the data obtained by the same sensor located in the same area, where i refers to the number of monitoring data sets, i∈{1, 2, 3······I}, and I refers to the maximum number of monitoring data sets; the number of monitoring data sets X i ={x i，1 , x i，2 , x i，3 ···x i，j ···x i，J}, where x i，j is the monitoring dataset X i The jth monitoring data in , j∈{1, 2, 3······J}, J is the monitoring data set X i The total number of monitoring data in x i，j =(t j , V i，j ), where t j is the monitoring data x i，j The corresponding time point, V i，j is the monitoring data x i，j Corresponding monitoring data value;

[0006] Select monitoring datasets X one by one i , for each monitoring dataset Xi Perform the following operations: Traverse the selected monitoring data set X i Monitoring data x in i，j , and according to the monitoring data x i，j The extreme point and time point t in j The monitoring dataset X i Divide into monitoring data value change segments;

[0007] According to the pairwise monitoring dataset X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust dataset;

[0008] Select trust datasets Q one by one i , and take the trust dataset Q i Use it as a training set to train the monitoring data matching model;

[0009] The monitoring data set X is matched with the monitoring data matching model in turn. i The monitoring data in the system are used for outlier detection.

[0010] As a preferred aspect of the present invention, according to the monitoring data x i，j The extreme point and time point t in j The monitoring data set is divided into monitoring data value change segments, specifically including the following steps: traverse the monitoring data set X i Monitoring data x i，j The corresponding monitoring data value V i，j , and according to the monitoring data value V i，j Get the extreme point x i，k , where k∈{1, 2, 3······J}, extreme points refer to monitoring data that are all greater than the adjacent monitoring data values ​​or are all less than the adjacent monitoring data values. The time points t corresponding to all extreme points are k Arrange them in order from small to large, and each adjacent time point t k The monitoring data value change segment T g , g is used to record the number of monitoring data value change segments, g∈{1, 2, 3······G}, G is the monitoring data set X i The total number of corresponding monitoring data value change segments; monitoring data value change segment T g The form is (t min , t max ), where t min To monitor the data value change period T g The smallest time point, t max To monitor the data value change period T g The largest time point.

[0011] As a preferred aspect of the present invention, according to the pairwise monitoring data set X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust data set specifically includes the following steps:

[0012] S1: Let r = 1, r is used as the number to select the monitoring data set;

[0013] S2: Select monitoring dataset X r ;

[0014] S3: Let u = 1, where u is used as a number to select the monitoring data value change segment;

[0015] S4: Select monitoring dataset X r The u-th monitoring data value change segment T u , the monitoring data value change segment T u Match with all monitoring data value change segments in the rest of the monitoring data set and calculate the number of successful matches y;

[0016] S5: Determine whether "y≥Y" is true, where Y is the confidence threshold. If "y≥Y" is true, change the monitoring data value by segment T. u Store the monitored data value change segment in a temporary storage set and enter S6; if "y≥Y" is not true, directly enter S6;

[0017] S6: Determine whether "u<G" holds. If so, assign u+1 to u and return to S4. If not, proceed to S7.

[0018] S7: Select monitoring data value change segments one by one from the monitoring data value change segment temporary set, and store all monitoring data corresponding to the time points within the selected monitoring data value change segments into the trust data set Q r ;

[0019] S8: Determine whether "r < I" holds. If so, assign r + 1 to r and return to S2. If not, proceed to S9.

[0020] S9: Output the entire trust data set Q i .

[0021] As a preferred aspect of the present invention, the monitoring data set X is matched sequentially by the monitoring data matching model. i The monitoring data in the system are detected for outliers, which specifically includes the following steps: Select the monitoring data set X one by one i , to monitor the dataset X iThe first monitoring data in is taken as the initial value, the initial value is input into the monitoring data matching model, and the predicted value is output in sequence, recorded as P1, and the corresponding true value is recorded as P2. If "|P1-P2| / P2<μ" holds, u is the fluctuation threshold, and this true value is not an abnormal value; if "|P1-P2| / P2<μ" does not hold, this true value is an abnormal value.

[0022] As a preferred aspect of the present invention, when generating a monitoring data value change segment, T g When the change trend B value is used to monitor the change segment T of the data value g Mark, B∈{0,1}, used to represent the monitoring data value change segment T g The trend of the monitoring data value is min and t max If the monitoring data value change trend between t is an increasing trend, then B=1; if the time point is t min and t max If the monitoring data value change trend between is a decreasing trend, then B=0;

[0023] In step S4, when the monitoring data value changes to segment T u When matching with all the monitoring data value change segments in the rest of the monitoring data set, if the match is successful, determine whether the B values ​​corresponding to the two successfully matched monitoring data value change segments are consistent. If they are consistent, no operation is performed; if they are inconsistent, the match is regarded as a match failure, and the monitoring data within the corresponding time points of the two successfully matched monitoring data value change segments are regarded as abnormal values.

[0024] As a preferred aspect of the present invention, the monitoring data matching model is established based on the RNN model.

[0025] A multi-time series monitoring data anomaly detection system, comprising:

[0026] A monitoring data set acquisition module is used to acquire monitoring data sets;

[0027] A monitoring data value change segment division module is used to divide the monitoring data set into monitoring data value change segments according to extreme value points and time points in the monitoring data;

[0028] A trust data set establishment module is used to determine the trust data set corresponding to the monitoring data set according to the monitoring data value change segment between two monitoring data sets;

[0029] A monitoring data matching model establishment module is used to establish a monitoring data matching model based on the trust data set corresponding to the monitoring data set;

[0030] The outlier detection module is used to perform outlier detection on the monitoring data in the monitoring data set in turn through the monitoring data matching model.

[0031] The present invention has the following advantages:

[0032] 1. The present invention deploys multiple sensors and regards the monitoring data with consistent change trends in different monitoring data sets of multiple time series as normal data. Based on these fragments of normal data, a matching model is established to identify abnormal values ​​in the monitoring data set. This can more accurately combine the characteristics of multiple time series data and has a high recognition accuracy.

[0033] 2. The present invention matches the changing trends of the monitoring data values ​​and regards the monitoring data with inconsistent changing trends as abnormal values, thereby avoiding the data errors caused by sensor failures affecting subsequent data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a schematic diagram of the structure of the multi-time series monitoring data anomaly detection system used in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0036] Example 1

[0037] A method for detecting anomalies in monitoring data of multiple time series, comprising:

[0038] Get monitoring dataset X i , is used to refer to the data obtained by the same sensor located in the same area, where i refers to the number of monitoring data sets, i∈{1, 2, 3······I}, and I refers to the maximum number of monitoring data sets; the number of monitoring data sets X i ={x i，1 , x i，2 , x i，3 ···x i，j ···x i，J}, where x i，j is the monitoring dataset X i The jth monitoring data in , j∈{1, 2, 3······J}, J is the monitoring data set X i The total number of monitoring data in x i，j =(t j , V i，j ), where t j is the monitoring data x i，j The corresponding time point, V i，j is the monitoring data x i，j Corresponding monitoring data value;

[0039] Select monitoring datasets X one by one i , for each monitoring dataset X i Perform the following operations: Traverse the selected monitoring data set X i Monitoring data x in i，j , and according to the monitoring data x i，j The extreme point and time point t in j The monitoring dataset X i Divide into monitoring data value change segments;

[0040] According to the monitoring data x i，j The extreme point and time point t in j The monitoring data set is divided into monitoring data value change segments, specifically including the following steps: traverse the monitoring data set X i Monitoring data x i，j The corresponding monitoring data value V i，j , and according to the monitoring data value V i，j Get the extreme point x i，k , where k∈{1, 2, 3······J}, extreme points refer to monitoring data that are all greater than or less than the adjacent monitoring data values. For example, a string of monitoring data values ​​is {0.53, 0.67, 0.69, 0.64, 0.55}, where the extreme points are the monitoring data corresponding to 0.53, 0.69, and 0.55. The time points t corresponding to all extreme points are k Arrange them in order from small to large, and each adjacent time point t k The monitoring data value change segment T g , g is used to record the number of monitoring data value change segments, g∈{1, 2, 3······G}, G is the monitoring data set X i The total number of corresponding monitoring data value change segments; for example, the time points corresponding to the extreme points are t2, t5, t8 and t 11 , the corresponding monitoring data change segments are (t2, t5), (t5, t8) and (t8, t 11 ); monitoring data value change period T g The form is (t min , t max ), where t min To monitor the data value change period T g The smallest time point, t max To monitor the data value change period T g The largest time point in

[0041] According to the pairwise monitoring dataset X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust dataset;

[0042] According to the pairwise monitoring dataset X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust data set specifically includes the following steps:

[0043] S1: Let r = 1, r is used as the number to select the monitoring data set;

[0044] S2: Select monitoring dataset X r ;

[0045] S3: Let u = 1, where u is used as a number to select the monitoring data value change segment;

[0046] S4: Select monitoring dataset X r The u-th monitoring data value change segment T u , the monitoring data value change segment T u Match with all monitoring data value change segments in the rest of the monitoring data set and calculate the number of successful matches y;

[0047] S5: Determine whether "y≥Y" is true, where Y is the confidence threshold, set by the operator, generally (0.7*I). If "y≥Y" is true, it means that the monitoring data value change segment T u The change of the monitoring data value within the period T has a high confidence level. u Store the monitoring data value change segment in the temporary storage set and enter S6; if "y≥Y" is not true, it means that the monitoring data value change segment T u If the change in the internal monitoring data value does not have a high confidence level, it will directly enter S6;

[0048] S6: Determine whether "u<G" is true. If "u<G" is true, it means that the monitoring data set X r The monitoring data value change segment in is not traversed completely, assign u+1 to u, and return to S4; if "u<G" does not hold, it means that the monitoring data set X r The monitoring data value change segment has been traversed completely, and enter S7;

[0049] S7: Select monitoring data value change segments one by one from the monitoring data value change segment temporary set, and store all monitoring data corresponding to the time points within the selected monitoring data value change segments into the trust data set Q r , trust dataset Q r The monitoring data in the sensor can be verified by other sensors, which means that the monitoring data is less likely to be affected by factors such as network transmission failure or sensor failure, and has higher data value;

[0050] S8: Determine whether "r < I" holds. If so, it means that not all monitoring data sets have been traversed. Assign r+1 to r and return to S2. If not, it means that all monitoring data sets have been traversed. Enter S9.

[0051] S9: Output the entire trust data set Q i ;

[0052] Select trust datasets Q one by one i , and take the trust dataset Q i The monitoring data matching model is trained as a training set, and the monitoring data matching model is established based on the RNN model.

[0053] The monitoring data set X is matched with the monitoring data matching model in turn. i The monitoring data within the system is detected for outliers. Here, the anomaly refers to the monitoring data value that is significantly different from the true value due to network transmission reasons or sensor failure.

[0054] The monitoring data set X is matched with the monitoring data matching model in turn. i The monitoring data in the system are detected for outliers, which specifically includes the following steps: Select the monitoring data set X one by one i , to monitor the dataset X i The first monitoring data is used as the initial value. This is because when the sensor is first used, it is considered to be trustworthy data because there are operators on site to debug it. The initial value is input into the monitoring data matching model, and the predicted value is output in sequence, recorded as P1, and the corresponding true value is recorded as P2. If "|P1-P2| / P2<μ" holds, u is the fluctuation threshold set by the operator, which means that the predicted value is not much different from the true value, and this true value is not an abnormal value; if "|P1-P2| / P2<μ" does not hold, it means that the predicted value is significantly different from the true value, and this true value is an abnormal value;

[0055] In this application, each monitoring data set is a time series data set. When data monitoring is carried out in a certain area, the deployed sensors may be affected by failures such as network transmission, resulting in abnormal values ​​in the monitoring data set. This application deploys multiple sensors and regards the monitoring data with consistent change trends in different monitoring data sets of multiple time series as normal data, and establishes a matching model based on these fragments of normal data to identify abnormal values ​​in the monitoring data set. It can more accurately combine the characteristics of multiple time series data and has a high recognition accuracy.

[0056] When generating the monitoring data value change segment T g When the change trend B value is used to monitor the change segment T of the data value g Mark, B∈{0,1}, used to represent the monitoring data value change segment Tg The trend of the monitoring data value is min and t max If the monitoring data value change trend between t is an increasing trend, then B=1; if the time point is t min and t max If the monitoring data value change trend between is a decreasing trend, then B=0;

[0057] In step S4, when the monitoring data value changes to segment T u When matching with all the monitoring data value change segments in the rest of the monitoring data sets, if the match is successful, determine whether the B values ​​corresponding to the two successfully matched monitoring data value change segments are consistent. If they are consistent, it means that the change trends corresponding to the two successfully matched monitoring data value change segments are consistent, and no operation is performed; if they are inconsistent, it means that the change trends corresponding to the two successfully matched monitoring data value change segments are inconsistent, and this match is regarded as a match failure, and the monitoring data within the corresponding time points of the two successfully matched monitoring data value change segments are regarded as abnormal values.

[0058] In this application, by matching the changing trends of the monitoring data values ​​and treating the monitoring data with inconsistent changing trends as abnormal values, the data errors caused by sensor failures are avoided from affecting subsequent data analysis.

[0059] Example 2

[0060] A multi-time series monitoring data anomaly detection system, such as Figure 1 Shown, including:

[0061] A monitoring data set acquisition module is used to acquire monitoring data sets;

[0062] A monitoring data value change segment division module is used to divide the monitoring data set into monitoring data value change segments according to extreme value points and time points in the monitoring data;

[0063] A trust data set establishment module is used to determine the trust data set corresponding to the monitoring data set according to the monitoring data value change segment between two monitoring data sets;

[0064] A monitoring data matching model establishment module is used to establish a monitoring data matching model based on the trust data set corresponding to the monitoring data set;

[0065] The outlier detection module is used to perform outlier detection on the monitoring data in the monitoring data set in turn through the monitoring data matching model.

[0066] It should be understood that those skilled in the art may make improvements or modifications based on the above description, and all such improvements and modifications shall fall within the scope of protection of the appended claims. Any portion of this specification not described in detail is prior art known to those skilled in the art.

Claims

1. A method for detecting anomalies in monitoring data of multiple time series, characterized in that: include: Get monitoring dataset X i , is used to refer to the data obtained by the same sensor located in the same area, where i refers to the number of monitoring data sets, i∈{1, 2, 3······I}, and I refers to the maximum number of monitoring data sets; the number of monitoring data sets X i ={x i，1 , x i，2 , x i，3 ···x i，j ···x i，J }, where x i，j is the monitoring dataset X i The jth monitoring data in , j∈{1, 2, 3······J}, J is the monitoring data set X i The total number of monitoring data in x i，j =(t j , V i，j ), where t j is the monitoring data x i，j The corresponding time point, V i，j is the monitoring data x i，j Corresponding monitoring data value; Select monitoring datasets X one by one i , for each monitoring dataset X i Perform the following operations: Traverse the selected monitoring data set X i Monitoring data x in i，j , and according to the monitoring data x i，j The extreme point and time point t in j The monitoring dataset X i Divide into monitoring data value change segments; According to the pairwise monitoring dataset X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust dataset; Select trust datasets Q one by one i , and take the trust dataset Q i Use it as a training set to train the monitoring data matching model; The monitoring data set X is matched with the monitoring data matching model in turn. i Perform outlier detection on the monitoring data within the system; According to the monitoring data x i，j The extreme point and time point t in j The monitoring data set is divided into monitoring data value change segments, specifically including the following steps: traverse the monitoring data set X i Monitoring data x i，j The corresponding monitoring data value V i，j , and according to the monitoring data value V i，j Get the extreme point x i，k , where k∈{1, 2, 3······J}, extreme points refer to monitoring data that are all greater than the adjacent monitoring data values ​​or are all less than the adjacent monitoring data values. The time points t corresponding to all extreme points are k Arrange them in order from small to large, and each adjacent time point t k The monitoring data value change segment T g , g is used to record the number of monitoring data value change segments, g∈{1, 2, 3······G}, G is the monitoring data set X i The total number of corresponding monitoring data value change segments; monitoring data value change segment T g The form is (t min , t max ), where t min To monitor the data value change period T g The smallest time point, t max To monitor the data value change period T g The largest time point in According to the pairwise monitoring dataset X i The monitoring data set X is determined by the monitoring data value change segment between i The corresponding trust data set specifically includes the following steps: S1: Let r = 1, r is used as the number to select the monitoring data set; S2: Select monitoring dataset X r ; S3: Let u=1, u is used as the number to select the monitoring data value change segment; S4: Select monitoring dataset X r The u-th monitoring data value change segment T u , the monitoring data value change segment T u Match with all monitoring data value change segments in the rest of the monitoring data set and calculate the number of successful matches y; S5: Determine whether "y≥Y" is true, where Y is the confidence threshold. If "y≥Y" is true, change the monitoring data value by T u Store the monitored data value change segment in a temporary storage set and enter S6; if "y≥Y" is not true, directly enter S6; S6: Determine whether "u<G" holds. If so, assign u+1 to u and return to S4. If not, proceed to S7. S7: Select monitoring data value change segments one by one from the monitoring data value change segment temporary set, and store all monitoring data corresponding to the time points within the selected monitoring data value change segments into the trust data set Q r ; S8: Determine whether "r<I" holds. If so, assign r+1 to r and return to S2. If not, proceed to S9. S9: Output the entire trust data set Q i .

2. The method for detecting anomalies in multi-time series monitoring data according to claim 1, characterized in that: The monitoring data set X is matched with the monitoring data matching model in turn. i The monitoring data in the system are detected for outliers, which specifically includes the following steps: Select the monitoring data set X one by one i , to monitor the dataset X i The first monitoring data in is taken as the initial value, the initial value is input into the monitoring data matching model, and the predicted value is output in sequence, recorded as P1, and the corresponding true value is recorded as P2. If "|P1-P2| / P2<μ" holds, u is the fluctuation threshold, and this true value is not an abnormal value; if "|P1-P2| / P2<μ" does not hold, this true value is an abnormal value.

3. The method for detecting anomalies in multi-time series monitoring data according to claim 2, characterized in that: When generating the monitoring data value change segment T g When the change trend B value is used to monitor the change segment T of the data value g Mark, B∈{0,1}, used to represent the monitoring data value change segment T g The trend of the monitoring data value is min and t max If the monitoring data value change trend between t is an increasing trend, then B=1; if the time point is t min and t max If the trend of the monitoring data value between is a decreasing trend, then B=0; In step S4, when the monitoring data value changes to segment T u When matching with all the monitoring data value change segments in the rest of the monitoring data set, if the match is successful, determine whether the B values ​​corresponding to the two successfully matched monitoring data value change segments are consistent. If they are consistent, no operation is performed; if they are inconsistent, the match is regarded as a match failure, and the monitoring data within the corresponding time points of the two successfully matched monitoring data value change segments are regarded as abnormal values.

4. The method for detecting anomalies in multi-time series monitoring data according to claim 3, wherein: The monitoring data matching model is established based on the RNN model.

5. A multi-time series monitoring data anomaly detection system, characterized in that: The system applies the multi-time series monitoring data anomaly detection method according to any one of claims 1 to 4, comprising: A monitoring data set acquisition module is used to acquire monitoring data sets; A monitoring data value change segment division module is used to divide the monitoring data set into monitoring data value change segments according to extreme value points and time points in the monitoring data; A trust data set establishment module is used to determine the trust data set corresponding to the monitoring data set according to the monitoring data value change segment between two monitoring data sets; A monitoring data matching model establishment module is used to establish a monitoring data matching model based on the trust data set corresponding to the monitoring data set; The outlier detection module is used to perform outlier detection on the monitoring data in the monitoring data set in turn through the monitoring data matching model.

Images