Abnormal data detection method, device, electronic device and computer storage medium

Abstract

The present application provides a method, device, electronic device and computer storage medium for detecting abnormal data, the method comprising: continuously acquiring the operation and maintenance data sequence at the current moment; then, for each operation and maintenance data sequence at the current moment, inputting the operation and maintenance data sequence at the current moment into a prediction model, and outputting the prediction model to obtain the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the prediction model is obtained by training the target codec model with a training sample set; the target codec model is a codec model with an attention mechanism added; then, according to the predicted value of the next minute of the data in all the operation and maintenance data sequences at the current moment, a dynamic baseline is generated; finally, if it is determined according to the dynamic baseline that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment, an alarm message is generated. Thus, the purpose of accurately and efficiently detecting abnormal data is achieved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and computer storage medium for detecting abnormal data. Background Art

[0002] At present, the performance data of application systems, such as transaction volume and response time, is a very important part of monitoring. Performance data is divided into different dimensions, such as provinces, cities, servers, etc.

[0003] With a large number of systems and multiple dimensions, the operation and maintenance manager needs to fully understand the situation of the systems he is responsible for. For basic data such as the maximum and minimum values ​​of historical data on a month-on-month basis, manually setting thresholds based on experience and historical month-on-month basis may not be accurate. At the same time, a large amount of setting work requires a lot of manpower costs. Summary of the invention

[0004] In view of this, the present application provides a method, device, electronic device and computer storage medium for detecting abnormal data, which are used to accurately and efficiently detect abnormal data.

[0005] The first aspect of the present application provides a method for detecting abnormal data, comprising:

[0006] Continuously obtain the current operation and maintenance data sequence;

[0007] For each operation and maintenance data sequence at the current moment, the operation and maintenance data sequence at the current moment is input into a prediction model, and the prediction model outputs a prediction value of the data in the operation and maintenance data sequence at the current moment for the next minute; wherein the prediction model is obtained by training a target codec model with a training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence;

[0008] Generate a dynamic baseline based on the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the dynamic baseline includes the actual value and the predicted value of each minute of the operation and maintenance data sequence;

[0009] According to the dynamic baseline, determine whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment;

[0010] If it is determined that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment, an alarm message is generated.

[0011] Optionally, the method for constructing the prediction model includes:

[0012] Constructing a training sample set; wherein the training sample set includes: at least one training sample data sequence; the training sample data sequence is based on a sampling rate of minutes, and N sampling points are used as a training sample data sequence;

[0013] Input the first N-1 sampling points in the training sample data sequence into the target encoding and decoding model, and output the predicted value of the Nth sampling point in the training sample data sequence;

[0014] According to the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence, the parameters in the target coding model are continuously adjusted until the error between the adjusted predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence meets a preset convergence condition, and the adjusted target coding model is determined as the prediction model.

[0015] Optionally, judging whether there is abnormal data in a target time period in the operation and maintenance data sequence at the current moment according to the dynamic baseline includes:

[0016] Determine the mean square error between the actual value and the predicted value in the operation and maintenance data sequence at the current moment in the target time period;

[0017] If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the operation and maintenance data sequence at the current moment.

[0018] Optionally, before determining the mean square error between the real value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period, the method further includes:

[0019] For each category of indicators in the operation and maintenance data sequence, a weight corresponding to the category is assigned to the indicator of the category to obtain a weighted operation and maintenance data sequence;

[0020] The step of determining the mean square error between the actual value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period includes:

[0021] Determine the mean square error between the actual value and the predicted value in the weighted operation and maintenance data sequence at the current time within the target time period;

[0022] Wherein, if the mean square error is greater than the reference standard, determining whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment includes:

[0023] If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the weighted operation and maintenance data sequence at the current moment.

[0024] Optionally, before determining the mean square error between the real value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period, the method further includes:

[0025] Normalizing the true value and the predicted value in the operation and maintenance data sequence at the current moment to obtain the normalized true value and the predicted value in the operation and maintenance data sequence at the current moment;

[0026] The step of determining the mean square error between the actual value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period includes:

[0027] Determine the mean square error between the actual value and the predicted value in the normalized current time operation and maintenance data sequence within the target time period;

[0028] Wherein, if the mean square error is greater than the reference standard, determining whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment includes:

[0029] If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the normalized operation and maintenance data sequence at the current moment.

[0030] Optionally, if it is determined that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment, after generating the alarm information, the method further includes:

[0031] Calculate the historical mean and the average of the historical proportion values ​​of each sampling point according to the historical data in the dynamic baseline;

[0032] The plus or minus 5 times the standard deviation of the historical mean and the historical proportion value of the sampling point are used as the upper and lower bounds of the mean;

[0033] Within a first preset time after the alarm information is generated, determining whether there are sampling points exceeding the upper and lower bounds within the first preset time;

[0034] If it is determined that there are sampling points exceeding the upper and lower limits within the first preset time, the alarm information is confirmed to be valid.

[0035] Optionally, if it is determined that there are sampling points exceeding the upper and lower bounds within the preset time, after confirming that the alarm information is valid, the method further includes:

[0036] The positive and negative three times standard deviation of the data mean of the second preset time before the current moment in the operation and maintenance data sequence at the current moment are used as the upper and lower bounds of the predicted value;

[0037] Determine whether the actual value at the current moment exceeds the upper and lower bounds of the predicted value;

[0038] If it is determined that the actual value at the current moment exceeds the upper and lower limits of the predicted value, an alarm prompt is generated.

[0039] A second aspect of the present application provides a device for detecting abnormal data, comprising:

[0040] An acquisition unit, used to continuously acquire the operation and maintenance data sequence at the current moment;

[0041] A prediction unit is used for inputting the operation and maintenance data sequence at the current moment into a prediction model for each operation and maintenance data sequence at the current moment, and obtaining the predicted value of the data in the operation and maintenance data sequence at the current moment in the next minute from the output of the prediction model; wherein the prediction model is obtained by training a target codec model with a training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence;

[0042] A first generating unit, configured to generate a dynamic baseline according to the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the dynamic baseline includes the actual value and the predicted value of each minute of the operation and maintenance data sequence;

[0043] A first judgment unit is used to judge whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment according to the dynamic baseline;

[0044] The second generating unit is used to generate alarm information if the first judging unit judges that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment.

[0045] Optionally, the construction unit of the prediction model includes:

[0046] A construction unit, used to construct a training sample set; wherein the training sample set includes: at least one training sample data sequence; the training sample data sequence is based on a sampling rate of minutes, and N sampling points are used as a training sample data sequence;

[0047] An input unit, used to input the first N-1 sampling points in the training sample data sequence into a target encoding and decoding model, and output a predicted value of the Nth sampling point in the training sample data sequence;

[0048] An adjustment unit is used to continuously adjust the parameters in the target coding model according to the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence, until the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence after adjustment meets a preset convergence condition, and the adjusted target coding model is determined as the prediction model.

[0049] Optionally, the first judging unit includes:

[0050] A first determination unit is used to determine the mean square error between the real value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period;

[0051] The second determining unit is configured to determine that abnormal data exists within a target time period in the operation and maintenance data sequence at the current moment if the mean square error is greater than a reference standard.

[0052] Optionally, the abnormal data detection method further includes:

[0053] A weighting unit, for assigning a weight corresponding to each category of indicators in the operation and maintenance data sequence to the indicators of the category, so as to obtain a weighted operation and maintenance data sequence;

[0054] Wherein, the first determining unit is used to:

[0055] Determine the mean square error between the actual value and the predicted value in the weighted operation and maintenance data sequence at the current time within the target time period;

[0056] Wherein, the second determining unit is used to:

[0057] If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the weighted operation and maintenance data sequence at the current moment.

[0058] Optionally, the abnormal data detection method further includes:

[0059] A normalization unit, used to perform normalization processing on the true value and the predicted value in the operation and maintenance data sequence at the current moment to obtain the normalized true value and the predicted value in the operation and maintenance data sequence at the current moment;

[0060] Wherein, the first determining unit is used to:

[0061] Determine the mean square error between the actual value and the predicted value in the normalized current time operation and maintenance data sequence within the target time period;

[0062] Wherein, the second determining unit is used to:

[0063] If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the normalized operation and maintenance data sequence at the current moment.

[0064] Optionally, the abnormal data detection method further includes:

[0065] A calculation unit, used for calculating the historical mean and the mean of the historical proportion value of each sampling point according to the historical data in the dynamic baseline;

[0066] A third determining unit is used to use plus or minus 5 times the standard deviation of the historical mean and the historical proportion value of the sampling point as the upper and lower bounds of the mean;

[0067] A second judgment unit, configured to judge whether there are sampling points exceeding the upper and lower bounds within a first preset time after the alarm information is generated;

[0068] A confirmation unit is used to confirm that the alarm information is valid if the second judgment unit determines that there are sampling points exceeding the upper and lower limits within the first preset time.

[0069] Optionally, the abnormal data detection method further includes:

[0070] A fourth determination unit is used to use the plus or minus three times standard deviation of the data mean of the second preset time before the current moment in the operation and maintenance data sequence at the current moment as the upper and lower bounds of the predicted value;

[0071] A third judgment unit, used to judge whether the actual value at the current moment exceeds the upper and lower bounds of the predicted value;

[0072] The third generating unit is used to generate an alarm prompt if the third judging unit judges that the actual value at the current moment exceeds the upper and lower limits of the predicted value.

[0073] A third aspect of the present application provides an electronic device, including:

[0074] one or more processors;

[0075] a storage device having one or more programs stored thereon;

[0076] When the one or more programs are executed by the one or more processors, the one or more processors implement the method for detecting abnormal data as described in any one of the first aspects.

[0077] A fourth aspect of the present application provides a computer storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method for detecting abnormal data as described in any one of the first aspects is implemented.

[0078] As can be seen from the above scheme, the present application provides a method, device, electronic device and computer storage medium for detecting abnormal data, the abnormal data detection method comprising: continuously acquiring the operation and maintenance data sequence at the current moment; then, for each operation and maintenance data sequence at the current moment, inputting the operation and maintenance data sequence at the current moment into the prediction model, and outputting the prediction model to obtain the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the prediction model is obtained by training the target codec model with a training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence; then, according to the predicted value of the next minute of the data in all the operation and maintenance data sequences at the current moment, a dynamic baseline is generated; wherein the dynamic baseline includes the true value and predicted value of each minute of the operation and maintenance data sequence; finally, according to the dynamic baseline, whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment; if it is determined that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment, an alarm message is generated. Thus, the purpose of accurately and efficiently detecting abnormal data is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0080] Figure 1 A specific flow chart of a method for detecting abnormal data provided in an embodiment of the present application;

[0081] Figure 2 A flowchart of a method for constructing a prediction model provided in another embodiment of the present application;

[0082] Figure 3 A schematic diagram of a network construction of a prediction model provided in another embodiment of the present application;

[0083] Figure 4 A flowchart of a method for detecting abnormal data provided by another embodiment of the present application;

[0084] Figure 5 A flowchart of a method for detecting abnormal data provided by another embodiment of the present application;

[0085] Figure 6 A flowchart of a method for detecting abnormal data provided by another embodiment of the present application;

[0086] Figure 7 A schematic diagram of a device for detecting abnormal data provided by another embodiment of the present application;

[0087] Figure 8 A schematic diagram of an electronic device for implementing a method for detecting abnormal data provided in another embodiment of the present application. DETAILED DESCRIPTION

[0088] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0089] It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units, and the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0090] The present application embodiment provides a method for detecting abnormal data, such as Figure 1 As shown, the specific steps include:

[0091] S101. Continuously obtain the operation and maintenance data sequence at the current moment.

[0092] S102. For each operation and maintenance data sequence at the current moment, the operation and maintenance data sequence at the current moment is input into a prediction model, and the prediction model outputs a prediction value of the data in the operation and maintenance data sequence at the current moment for the next minute.

[0093] The prediction model is obtained by training the target encoding and decoding model with a training sample set; the target encoding and decoding model is an encoder-decoder model with an attention mechanism added; the training sample set includes: at least one training sample data sequence.

[0094] It should be noted that the existing Encoder-Decode model inputs a fixed-length time series into the Encoder, and the output of the existing Encoder-Decode model is a fixed-length vector. The Encoder maps the input into a potential representation, and the Decoder decodes the potential representation to restore it to its original form. In theory, the Encoder-Decode structure is a lossy compression method and has a noise reduction function, so it has the ability to filter outliers or abnormal sequences.

[0095] However, the Encoder-Decode model also has certain limitations. The biggest limitation is that the only connection between the encoder and the decoder is a fixed-length vector. That is, the encoder has to compress the information of the entire sequence into a fixed-length vector. Obviously, this vector cannot fully represent the information of the entire sequence, and the information carried by the first input content will be diluted or covered by the later input information. As the length of the input sequence increases, this phenomenon will become more and more serious. If an input sequence without sufficient information is obtained at the initial stage of decoding, the accuracy of decoding will naturally decrease.

[0096] Therefore, this application introduces the Attention mechanism. The basic idea of ​​the Attention mechanism is to break the limitation that the traditional Encoder-Decoder structure relies on a fixed-length internal vector during encoding. When the Encoder encodes the input sentence, we retain the hidden state vector corresponding to each moment in the sequence, and when the Decoder decodes, we make full use of this information (hidden state vector) instead of just relying on the last hidden state vector output by the Encoder.

[0097] The Attention mechanism is a method of selectively focusing on some information from a large amount of existing information based on a certain current state. It retains the intermediate results of the encoder encoding the input sequence, and then trains a model to selectively learn these inputs and associate the output sequence with them when outputting. Instead of using a fixed vector representation like the traditional Encoder-Decoder, the model selectively focuses on the corresponding relevant information in the input when outputting the model.

[0098] Optionally, in another embodiment of the present application, an implementation of a method for constructing a prediction model is as follows: Figure 2 As shown, including:

[0099] S201: Construct a training sample set.

[0100] The training sample set includes: at least one training sample data sequence; the training sample data sequence is sampled at a minute-level sampling rate, and N sampling points are used as a training sample data sequence.

[0101] S202: Input the first N-1 sampling points in the training sample data sequence into the target encoding and decoding model, and output the predicted value of the Nth sampling point in the training sample data sequence.

[0102] S203: Determine whether the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence meets a preset convergence condition.

[0103] Specifically, if it is determined that the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence meets the preset convergence condition, step S204 is executed; if it is determined that the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence does not meet the preset convergence condition, step S205 is executed.

[0104] S204: Determine the target coding model as the prediction model.

[0105] S205: Adjust parameters in the target coding model.

[0106] like Figure 3 As shown in Figure 1, it is a schematic diagram of the network construction of the prediction model. Figure 3 Let's take an example:

[0107] First, select three months of historical data for training. According to the sampling rate of minutes, that is, one monitoring indicator per minute, 1440 monitoring indicators per day, a total of about 1440*30*3 monitoring indicator data in time sequence. Grouping is done in groups of 50 points, that is, the first to the 50th point is a group, the second to the 51st point is a group, and so on. In each group of data, we input the first 49 points into the LSTM network. The lossy filtering of data is achieved by adding Encoder-Decode and corresponding attention between each layer of the LSTM network. The loss of data enhances the training effect of the data. Finally, the predicted value is obtained, and the predicted value is compared with the 50th actual data of the group to determine the accuracy of the prediction. Through cyclic training, a model is formed.

[0108] S103: Generate a dynamic baseline based on the predicted value of the next minute of data in all the operation and maintenance data sequences at the current moment.

[0109] The dynamic baseline includes the actual value and predicted value of the operation and maintenance data series every minute.

[0110] Continuing with the above example, in actual applications, by inputting the time series indicator data within the current 49 minutes into the prediction model, the data for the next minute can be immediately obtained, and the latest data obtained every minute can be predicted in real time and continuously, thus obtaining a dynamic baseline consisting of predicted data and real data.

[0111] S104: According to the dynamic baseline, determine whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment.

[0112] Specifically, if it is determined that abnormal data exists in the target time period in the operation and maintenance data sequence at the current moment, step S105 is executed.

[0113] S105: Generate warning information.

[0114] Optionally, in another embodiment of the present application, an implementation of step S104 is as follows: Figure 4 As shown, including:

[0115] S401. Determine the mean square error between the real value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period.

[0116] The target time period can be preset and changed by technical personnel or relevant authorized staff, such as 5 minutes, 3 minutes, 10 minutes, etc., and is not limited here.

[0117] Specifically, the mean square error between the actual value and the predicted value in the operation and maintenance data sequence at the current moment can be calculated using the following calculation formula:

[0118]

[0119] Among them, x i is the true value of the i-th minute, y i is the predicted value of the i-th minute, and n is the target time period.

[0120] S402: If the mean square error is greater than the reference standard, it is determined that abnormal data exists in the target time period in the operation and maintenance data sequence at the current moment.

[0121] Among them, the reference standard is that in the early stage of training the prediction model, the mean square error of the data in the training set every 5 minutes can be calculated, and the largest mean square error value is selected as the reference standard.

[0122] In the actual operation and maintenance process, the tolerance of the upper and lower thresholds of different types of indicators is different. For example, for the trading volume indicator, when the actual value exceeds the predicted value, although an alarm is generated, it is judged as a false alarm due to the instantaneous surge in transactions, holidays, etc. However, when the trading volume is lower than the predicted value, it often indicates the occurrence of an abnormality. Therefore, in another embodiment of the present application, before executing step S401, it also includes:

[0123] For each category of indicators in the operation and maintenance data sequence, a weight corresponding to the category is assigned to the category indicator to obtain a weighted operation and maintenance data sequence.

[0124] Among them, the upper and lower weight coefficients corresponding to different indicators can be shown in Table 1.

[0125] Indicator Type Upper weight Lower weight Trading Volume 0.6 1 Success rate 0.5 1 Response time 1 0.5

[0126] Table 1

[0127] It should be noted that the weight coefficient can be adjusted according to actual conditions. For example, the person in charge of a system feels that a surge in trading volume should not be an alarm and that it is a normal business situation. However, if the surge is particularly severe and an alarm is required, then when the actual value minus the predicted value is greater than 0, we will multiply it by the weight. For example, a surge in trading volume of 10,000 will ultimately only be counted as a surge of 6,000 to ensure that no false alarms are issued in this situation.

[0128] Then the corresponding step S401 is to determine the mean square error between the true value and the predicted value in the weighted operation and maintenance data sequence at the current moment within the target time period; step S402 is to determine whether there is abnormal data in the weighted operation and maintenance data sequence at the current moment within the target time period if the mean square error is greater than the reference standard.

[0129] For the trading volume indicator, if the trading volume per minute is large, the calculated mean square error indicator fluctuates greatly as a whole, which is not conducive to data comparison. Therefore, the data needs to be scaled to between 0 and 1 for normalization. Therefore, in another embodiment of the present application, before executing step S401, it also includes:

[0130] The true value and the predicted value in the operation and maintenance data sequence at the current moment are normalized to obtain the normalized true value and the predicted value in the operation and maintenance data sequence at the current moment.

[0131] Specifically, the following normalization processing method can be used:

[0132] f(x i )=(x i -min(x)) / (max(x)-min(x));

[0133] Among them, x iIn order to normalize the real value, considering the timeliness of the alarm, this solution requires that the anomaly be detected within the target time period (such as 5 minutes). Therefore, for the time series performance data with one sampling point per minute, we need to normalize the data of the five points i=1, 2, 3, 4, and 5. max(x) and min(x) are the maximum value of the actual value and the predicted value within 5 minutes as the maximum value in the formula, and the minimum value as the minimum value in the formula. f(x i ) is the actual value x i Normalized results.

[0134] Then the corresponding step S401 is to determine the mean square error between the true value and the predicted value in the normalized operation and maintenance data sequence at the current moment within the target time period; step S402 is to determine whether there is abnormal data in the normalized operation and maintenance data sequence at the current moment within the target time period if the mean square error is greater than the reference standard.

[0135] Optionally, in another embodiment of the present application, an implementation after step S105 is as follows: Figure 5 As shown, it also includes:

[0136] S501. Calculate the historical mean and the mean of the historical proportion values ​​of each sampling point according to the historical data in the dynamic baseline.

[0137] S502: Take the historical mean of the sampling points and the mean of the historical proportion values ​​plus or minus 5 times the standard deviation as the upper and lower bounds of the mean.

[0138] S503: Within a first preset time after the alarm information is generated, determine whether there are sampling points exceeding the upper and lower bounds within the first preset time.

[0139] Specifically, if it is determined that there are sampling points exceeding the upper and lower bounds within the first preset time, step S504 is executed.

[0140] S504: Confirm that the warning information is valid.

[0141] Specifically, the alarm range is determined by the historical average and ratio values ​​to determine whether the alarm information is within a reasonable range, and a portion of false alarms are filtered out again.

[0142] In the case of low data values, if the overall fluctuation between the predicted value and the true value is not large, a false alarm will still be generated. Therefore, in another embodiment of the present application, an implementation method after step S504 is as follows: Figure 6 As shown, it also includes:

[0143] S601. Take the plus or minus three times standard deviation of the data mean of the second preset time before the current moment in the operation and maintenance data sequence at the current moment as the upper and lower bounds of the predicted value.

[0144] S602: Determine whether the actual value at the current moment exceeds the upper and lower bounds of the predicted value.

[0145] Specifically, if it is determined that the actual value at the current moment exceeds the upper and lower bounds of the predicted value, step S603 is executed.

[0146] S603: Generate an alarm prompt.

[0147] As can be seen from the above scheme, the present application provides a method for detecting abnormal data: continuously acquiring the operation and maintenance data sequence at the current moment; then, for each operation and maintenance data sequence at the current moment, inputting the operation and maintenance data sequence at the current moment into the prediction model, and outputting the prediction model to obtain the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the prediction model is obtained by training the target codec model with the training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence; then, according to the predicted value of the next minute of the data in all the operation and maintenance data sequences at the current moment, a dynamic baseline is generated; wherein the dynamic baseline includes the true value and the predicted value of each minute of the operation and maintenance data sequence; finally, according to the dynamic baseline, it is determined whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment; if it is determined that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment, an alarm message is generated. Thus, the purpose of accurately and efficiently detecting abnormal data is achieved.

[0148] Another embodiment of the present application provides a device for detecting abnormal data, such as Figure 7 As shown, specifically including:

[0149] The acquisition unit 701 is used to continuously acquire the operation and maintenance data sequence at the current moment.

[0150] The prediction unit 702 is used to input the operation and maintenance data sequence at the current moment into the prediction model for each operation and maintenance data sequence at the current moment, and obtain the prediction value of the data in the operation and maintenance data sequence at the current moment in the next minute from the output of the prediction model.

[0151] The prediction model is obtained by training the target codec model with a training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence.

[0152] Optionally, in another embodiment of the present application, an implementation of a construction unit of a prediction model includes:

[0153] Construction unit, used to construct training sample set.

[0154] The training sample set includes: at least one training sample data sequence; the training sample data sequence is sampled at a minute-level sampling rate, and N sampling points are used as a training sample data sequence.

[0155] The input unit is used to input the first N-1 sampling points in the training sample data sequence into the target encoding and decoding model, and output the predicted value of the Nth sampling point in the training sample data sequence.

[0156] The adjustment unit is used to continuously adjust the parameters in the target coding model according to the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence, until the error between the predicted value of the Nth sampling point in the training sample data sequence after adjustment and the true value of the Nth sampling point in the training sample data sequence meets the preset convergence condition, and the adjusted target coding model is determined as the prediction model.

[0157] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 2 As shown, no further description is given here.

[0158] The first generating unit 703 is used to generate a dynamic baseline according to the predicted value of the next minute of the data in all the operation and maintenance data sequences at the current moment.

[0159] The dynamic baseline includes the actual value and predicted value of the operation and maintenance data series every minute.

[0160] The first judgment unit 704 is used to judge whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment according to the dynamic baseline.

[0161] Optionally, in another embodiment of the present application, an implementation of the first judgment unit 704 includes:

[0162] The first determination unit is used to determine the mean square error between the real value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period.

[0163] The second determining unit is used to determine that abnormal data exists in the target time period in the operation and maintenance data sequence at the current moment if the mean square error is greater than a reference standard.

[0164] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 4 As shown, no further description is given here.

[0165] Optionally, in another embodiment of the present application, an implementation of the abnormal data detection device further includes:

[0166] A weighting unit is used to assign a weight corresponding to each category of indicators in the operation and maintenance data sequence to obtain a weighted operation and maintenance data sequence;

[0167] Wherein, the first determining unit is used to:

[0168] Determine the mean square error between the actual value and the predicted value in the weighted operation and maintenance data sequence at the current time within the target time period;

[0169] Wherein, the second determining unit is used for:

[0170] If the mean square error is greater than the reference standard, it is determined that there is abnormal data in the target time period in the weighted operation and maintenance data sequence at the current moment.

[0171] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be described in detail here.

[0172] Optionally, in another embodiment of the present application, an implementation of the abnormal data detection device further includes:

[0173] A normalization unit, used to normalize the real value and the predicted value in the operation and maintenance data sequence at the current moment, to obtain the normalized real value and the predicted value in the operation and maintenance data sequence at the current moment;

[0174] Wherein, the first determining unit is used to:

[0175] Determine the mean square error between the actual value and the predicted value in the normalized current time operation and maintenance data sequence within the target time period;

[0176] Wherein, the second determining unit is used for:

[0177] If the mean square error is greater than the reference standard, it is determined that there is abnormal data in the target time period in the normalized operation and maintenance data sequence at the current moment.

[0178] The specific working process of the units disclosed in the above embodiments of the present application can be found in the corresponding method embodiments, which will not be described in detail here.

[0179] The second generating unit 705 is configured to generate an alarm message if the first judging unit 704 judges that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment.

[0180] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 1 As shown, no further description is given here.

[0181] Optionally, in another embodiment of the present application, an implementation of the abnormal data detection device further includes:

[0182] The calculation unit is used to calculate the historical mean and the average of the historical proportion values ​​of each sampling point according to the historical data in the dynamic baseline.

[0183] The third determining unit is used to take plus or minus 5 times the standard deviation of the historical mean value and the mean value of the historical proportion value of the sampling point as the upper and lower bounds of the mean value.

[0184] The second judgment unit is used to judge whether there are sampling points exceeding the upper and lower limits within the first preset time after the alarm information is generated.

[0185] The confirmation unit is used to confirm that the alarm information is valid if the second judgment unit determines that there are sampling points exceeding the upper and lower limits within the first preset time.

[0186] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 5 As shown, no further description is given here.

[0187] Optionally, in another embodiment of the present application, an implementation of the abnormal data detection device further includes:

[0188] The fourth determination unit is used to use the plus or minus three times standard deviation of the data mean of the second preset time before the current moment in the operation and maintenance data sequence at the current moment as the upper and lower bounds of the predicted value.

[0189] The third judgment unit is used to judge whether the actual value at the current moment exceeds the upper and lower limits of the predicted value.

[0190] The third generating unit is used to generate an alarm prompt if the third judging unit judges that the actual value at the current moment exceeds the upper and lower limits of the predicted value.

[0191] For the specific working process of the units disclosed in the above embodiments of the present application, please refer to the corresponding method embodiments, such as Figure 6 As shown, no further description is given here.

[0192] As can be seen from the above scheme, the present application provides a detection device for abnormal data: the acquisition unit 701 continuously acquires the operation and maintenance data sequence at the current moment; then, the prediction unit 702 inputs the operation and maintenance data sequence at the current moment into the prediction model for each operation and maintenance data sequence at the current moment, and the prediction model outputs the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the prediction model is obtained by training the target codec model with the training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence; the first generation unit 703 generates a dynamic baseline according to the predicted value of the next minute of the data in all the operation and maintenance data sequences at the current moment; wherein the dynamic baseline includes the true value and the predicted value of each minute of the operation and maintenance data sequence; finally, the first judgment unit 704 judges whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment according to the dynamic baseline; if the first judgment unit 704 judges that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment, the second generation unit 705 generates an alarm message. Thus, the purpose of accurately and efficiently detecting abnormal data is achieved.

[0193] Another embodiment of the present application provides an electronic device, such as Figure 8 As shown, including:

[0194] One or more processors 801.

[0195] The storage device 802 stores one or more programs.

[0196] When the one or more programs are executed by the one or more processors 801, the one or more processors 801 implement the abnormal data detection method as described in any one of the above embodiments.

[0197] Another embodiment of the present application provides a computer storage medium on which a computer program is stored, wherein when the computer program is executed by a processor, the method for detecting abnormal data as described in any one of the above embodiments is implemented.

[0198] In the above embodiments disclosed in the present application, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0199] In addition, each functional module in each embodiment of the present disclosure can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a live broadcast device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0200] The present application can be implemented or used by professionals and technicians. Various modifications to these embodiments will be apparent to professionals and technicians in the field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for detecting abnormal data, characterized in that: include: Continuously obtain the operation and maintenance data sequence at the current moment; For each operation and maintenance data sequence at the current moment, the operation and maintenance data sequence at the current moment is input into a prediction model, and the prediction model outputs a prediction value of the data in the operation and maintenance data sequence at the current moment for the next minute; wherein the prediction model is obtained by training a target codec model with a training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence; Generate a dynamic baseline based on the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the dynamic baseline includes the actual value and the predicted value of each minute of the operation and maintenance data sequence; According to the dynamic baseline, determine whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment; If it is determined that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment, an alarm message is generated; Calculate the historical mean and the average of the historical proportion values ​​of each sampling point according to the historical data in the dynamic baseline; The plus or minus 5 times the standard deviation of the historical mean and the historical proportion value of the sampling point are used as the upper and lower bounds of the mean; Within a first preset time after the alarm information is generated, determining whether there are sampling points exceeding the upper and lower bounds within the first preset time; If it is determined that there are sampling points exceeding the upper and lower bounds within the first preset time, confirming that the alarm information is valid; The positive and negative three times standard deviation of the data mean of the second preset time before the current moment in the operation and maintenance data sequence at the current moment are used as the upper and lower bounds of the predicted value; Determine whether the actual value at the current moment exceeds the upper and lower bounds of the predicted value; If it is determined that the actual value at the current moment exceeds the upper and lower limits of the predicted value, an alarm prompt is generated.

2. The detection method according to claim 1, characterized in that: The method for constructing the prediction model comprises: Constructing a training sample set; wherein the training sample set includes: at least one training sample data sequence; the training sample data sequence is based on a sampling rate of minutes, and N sampling points are used as a training sample data sequence; Input the first N-1 sampling points in the training sample data sequence into the target encoding and decoding model, and output the predicted value of the Nth sampling point in the training sample data sequence; According to the error between the predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence, the parameters in the target coding model are continuously adjusted until the error between the adjusted predicted value of the Nth sampling point in the training sample data sequence and the true value of the Nth sampling point in the training sample data sequence meets a preset convergence condition, and the adjusted target coding model is determined as the prediction model.

3. The detection method according to claim 1, characterized in that: The determining, based on the dynamic baseline, whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment includes: Determine the mean square error between the actual value and the predicted value in the operation and maintenance data sequence at the current moment in the target time period; If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the operation and maintenance data sequence at the current moment.

4. The detection method according to claim 3, characterized in that: Before determining the mean square error between the real value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period, the method further includes: For each category of indicators in the operation and maintenance data sequence, a weight corresponding to the category is assigned to the indicator of the category to obtain a weighted operation and maintenance data sequence; The step of determining the mean square error between the actual value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period includes: Determine the mean square error between the actual value and the predicted value in the weighted operation and maintenance data sequence at the current time within the target time period; Wherein, if the mean square error is greater than the reference standard, determining whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment includes: If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the weighted operation and maintenance data sequence at the current moment.

5. The detection method according to claim 3, characterized in that: Before determining the mean square error between the real value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period, the method further includes: Normalizing the true value and the predicted value in the operation and maintenance data sequence at the current moment to obtain the normalized true value and the predicted value in the operation and maintenance data sequence at the current moment; The step of determining the mean square error between the actual value and the predicted value in the operation and maintenance data sequence at the current moment within the target time period includes: Determine the mean square error between the actual value and the predicted value in the normalized current time operation and maintenance data sequence within the target time period; Wherein, if the mean square error is greater than the reference standard, determining whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment includes: If the mean square error is greater than the reference standard, it is determined that abnormal data exists within the target time period in the normalized operation and maintenance data sequence at the current moment.

6. A device for detecting abnormal data, characterized in that: include: An acquisition unit, used to continuously acquire the operation and maintenance data sequence at the current moment; A prediction unit is used for inputting the operation and maintenance data sequence at the current moment into a prediction model for each operation and maintenance data sequence at the current moment, and obtaining the predicted value of the data in the operation and maintenance data sequence at the current moment in the next minute from the output of the prediction model; wherein the prediction model is obtained by training a target codec model with a training sample set; the target codec model is a codec model with an attention mechanism added; the training sample set includes: at least one training sample data sequence; A first generating unit, configured to generate a dynamic baseline according to the predicted value of the next minute of the data in the operation and maintenance data sequence at the current moment; wherein the dynamic baseline includes the actual value and the predicted value of each minute of the operation and maintenance data sequence; A first judgment unit is used to judge whether there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment according to the dynamic baseline; A second generating unit is configured to generate an alarm message if the first judging unit judges that there is abnormal data in the target time period in the operation and maintenance data sequence at the current moment; A calculation unit, used for calculating the historical mean and the mean of the historical proportion value of each sampling point according to the historical data in the dynamic baseline; A third determining unit is used to use plus or minus 5 times the standard deviation of the historical mean and the historical proportion value of the sampling point as the upper and lower bounds of the mean; A second judgment unit, configured to judge whether there are sampling points exceeding the upper and lower bounds within a first preset time after the alarm information is generated; A confirmation unit, configured to confirm that the alarm information is valid if it is determined that there are sampling points exceeding the upper and lower bounds within the first preset time; A fourth determination unit is used to use the plus or minus three times standard deviation of the data mean of the second preset time before the current moment in the operation and maintenance data sequence at the current moment as the upper and lower bounds of the predicted value; A third judgment unit, used to judge whether the actual value at the current moment exceeds the upper and lower bounds of the predicted value; The third generating unit is used to generate an alarm prompt if it is determined that the actual value at the current moment exceeds the upper and lower limits of the predicted value.

7. An electronic device, characterized in that: include: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the abnormal data detection method as described in any one of claims 1 to 5.

8. A computer storage medium, characterized in that: A computer program is stored thereon, wherein when the computer program is executed by a processor, the method for detecting abnormal data as claimed in any one of claims 1 to 5 is implemented.

Images