A data detection method and apparatus, an electronic device, and a storage medium

Abstract

The application discloses a data detection method and device, electronic equipment and a storage medium. The method comprises the following steps: obtaining to-be-detected data, wherein the to-be-detected data is first performance index time series data; inputting the to-be-detected data into a convolutional neural network (CNN) and a long short-term memory network (LSTM) respectively to obtain first feature data and second feature data; obtaining a data baseline of the to-be-detected data based on the first feature data and the second feature data; and determining abnormal data corresponding to the to-be-detected data by using the data baseline.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data detection method, apparatus, electronic device, and storage medium. Background Technology

[0002] To provide efficient and reliable services, internet companies must monitor the time-series performance metrics of their existing services and applications in real time to promptly detect anomalies. When monitoring performance metrics time-series data, even brief data fluctuations should be detected, as these could be dangerous warning signals. Currently, anomaly detection for performance metrics time-series data primarily relies on a coarse-grained approach using threshold rules, i.e., a fixed baseline method, where data exceeding a fixed baseline threshold is considered anomalous. While threshold rules are easy to understand and operate, enabling basic detection of performance metrics time-series data, they suffer from significant drawbacks, particularly low anomaly detection effectiveness. Low threshold values ​​result in numerous invalid detections, while high threshold values ​​lead to missed anomalies. Additionally, some fields employ machine learning methods for more in-depth anomaly detection in performance data, but their application in the performance metrics time-series data domain is still relatively limited, and their accuracy and effectiveness remain somewhat lacking. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a data detection method, apparatus, electronic device, and storage medium.

[0004] In a first aspect, embodiments of this application provide a data detection method, the method comprising:

[0005] Obtain the data to be tested, wherein the data to be tested is the time series data of the first performance index;

[0006] The data to be detected is input into a convolutional neural network (CNN) and a long short-term memory network (LSTM) to obtain the first feature data and the second feature data, respectively.

[0007] Based on the first feature data and the second feature data, the data baseline of the data to be detected is obtained;

[0008] The abnormal data corresponding to the data to be detected is determined using the data baseline.

[0009] In one optional embodiment of this application, obtaining the data to be detected includes:

[0010] The script program collects time-series data of the second performance indicator of the network system according to the set time granularity.

[0011] The time-series data of the second performance index is preprocessed to obtain the data to be tested.

[0012] In an optional embodiment of this application, the preprocessing of the time-series data of the second performance index to obtain the data to be tested includes:

[0013] The missing data in the time series data of the second performance index is filled in, and the time series data of the second performance index after filling in the missing data is standardized to obtain the data to be tested.

[0014] In one optional embodiment of this application, the data to be detected includes data in n dimensions, where n is an integer greater than or equal to 1; the step of inputting the data to be detected into a convolutional neural network (CNN) and a long short-term memory network (LSTM) respectively to obtain first feature data and second feature data includes:

[0015] The data to be detected is input into a CNN to obtain the features of the data to be detected in n dimensions; based on the features of the data to be detected in the n dimensions, the first feature data is obtained; the first feature data is an m1*n dimensional vector;

[0016] The data to be detected is input into an LSTM to obtain the features output by the n hidden layers of the LSTM; the dimension of each of the n hidden layers is m2; based on the features output by the n hidden layers, second feature data is obtained; the second feature data is an n*m2 dimensional vector.

[0017] In an optional embodiment of this application, obtaining the data baseline of the data to be detected based on the first feature data and the second feature data includes:

[0018] The first feature data and the second feature data are fused using an attention mechanism to obtain a first weight;

[0019] The first weight is normalized using a normalization function to obtain the second weight;

[0020] The output of the hidden layer at each time point in the LSTM is multiplied by the corresponding second weight in the second weight to obtain the third feature data;

[0021] The data baseline of the data to be detected is predicted based on the third feature data.

[0022] In an optional embodiment of this application, the step of using the data baseline to determine the abnormal data corresponding to the data to be detected includes:

[0023] Identify m3 peaks that are higher than a set threshold among the multiple peaks included in the data baseline;

[0024] Based on the times corresponding to the m3 peak values, m3 detection windows are determined;

[0025] The overlapping region of the m3 detection windows is determined, and the second performance index time series data with a value greater than the maximum value among the m3 peaks in the overlapping region is identified as abnormal data.

[0026] In one optional embodiment of this application, determining the m3 detection windows based on the times corresponding to the m3 peaks includes:

[0027] For each of the m3 peaks, the region corresponding to the target time period corresponding to that time is determined as the detection window corresponding to that peak; the target time period is the time period included before and after that time period, which is a predetermined duration.

[0028] Secondly, embodiments of this application provide a data detection device, the device comprising:

[0029] An acquisition unit is used to acquire data to be detected, wherein the data to be detected is time-series data of a first performance index.

[0030] The first processing unit is used to input the data to be detected into the convolutional neural network (CNN) and the long short-term memory network (LSTM) respectively to obtain the first feature data and the second feature data.

[0031] The second processing unit is used to obtain the data baseline of the data to be detected based on the first feature data and the second feature data;

[0032] The determination unit is used to determine abnormal data corresponding to the data to be detected using the data baseline.

[0033] In one optional embodiment of this application, the obtaining unit is specifically used to: collect time-series data of the second performance index of the network system according to a set time granularity using a script program; and preprocess the time-series data of the second performance index to obtain data to be detected.

[0034] In an optional embodiment of this application, the obtaining unit is further specifically used to: fill in the missing data in the second performance index time series data, and perform standardization processing on the second performance index time series data after filling in the missing data to obtain the data to be detected.

[0035] In an optional embodiment of this application, the data to be detected includes data in n dimensions, where n is an integer greater than or equal to 1; the first processing unit is specifically used for: inputting the data to be detected into a CNN to obtain features of the data to be detected in n dimensions; obtaining first feature data based on the features of the data to be detected in the n dimensions; the first feature data is an m1*n dimensional vector; inputting the data to be detected into an LSTM to obtain features output by the data to be detected in the n hidden layers of the LSTM; the dimension of each of the n hidden layers is m2; obtaining second feature data based on the features output by the n hidden layers; the second feature data is an n*m2 dimensional vector.

[0036] In an optional embodiment of this application, the second processing unit is specifically used for: fusing the first feature data and the second feature data using an attention mechanism to obtain a first weight; normalizing the first weight using a normalization function to obtain a second weight; multiplying the output of the hidden layer at each time point included in the LSTM with the corresponding second weight in the second weight to obtain third feature data; and predicting the data baseline of the data to be detected based on the third feature data.

[0037] In an optional embodiment of this application, the determining unit is specifically used to: determine m3 peaks that are higher than a set threshold among the multiple peaks included in the data baseline; determine m3 detection windows based on the time corresponding to the m3 peaks; determine the overlapping area of ​​the m3 detection windows, and determine the second performance index time series data whose value is greater than the maximum value among the m3 peaks in the second performance index time series data corresponding to the overlapping area as abnormal data.

[0038] In an optional embodiment of this application, the determining unit is further specifically configured to: for each of the m3 peaks, determine the region corresponding to the target time period corresponding to that time period as the detection window corresponding to that peak period; the target time period is the time period included by a predetermined duration before and after that time period.

[0039] Thirdly, embodiments of this application provide an electronic device, which includes a memory and a processor. The memory stores computer-executable instructions, and the processor can implement the methods described in the above embodiments when it executes the computer-executable instructions in the memory.

[0040] Fourthly, embodiments of this application provide a computer storage medium storing executable instructions, which, when executed by a processor, implement the methods described in the above embodiments.

[0041] The technical solution of this application fully utilizes the characteristics of Convolutional Neural Networks (CNN) in extracting features from multidimensional sequence data and the characteristics of Long Short-Term Memory (LSTM) networks in capturing the dependencies and making predictions of time series data. When using a hybrid neural network combining CNN and LSTM to extract features from the data to be detected, the extracted features can not only reflect the spatial relationships between the various dimensions of time series data of different performance indicators, but also reflect the changes of time series data of performance indicators in the time dimension. The accuracy of the data baseline predicted based on the extracted features is higher, and the reliability and effectiveness of the detected abnormal data are higher when using the predicted data baseline to detect abnormal data. Attached Figure Description

[0042] Figure 1 A flowchart illustrating the data detection method provided in this application embodiment;

[0043] Figure 2 A schematic diagram of the CNN and LSTM feature extraction process provided in the embodiments of this application;

[0044] Figure 3 A schematic diagram illustrating the feature fusion process using an attention mechanism provided in an embodiment of this application;

[0045] Figure 4 A schematic diagram illustrating the implementation of the data detection method provided in this application embodiment;

[0046] Figure 5 This is a schematic diagram of the structural composition of the data detection device provided in the embodiments of this application;

[0047] Figure 6 This is a schematic diagram of the structural composition of the electronic device provided in the embodiments of this application. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0049] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be defined and explained in subsequent figures.

[0050] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0051] Performance metrics time series data generally refers to some performance indicators and system status indicators during the operation of a network system. These are usually time series collected at fixed time intervals. These data will form some potential relationships and have some specific patterns. By analyzing these data, we can identify the current problems in the network system.

[0052] Ideal performance index time series data is smooth, without excessive noise and distortion, while real performance index time series data usually contains a lot of uncertain random noise. This random noise can easily be mistaken for anomalous data when performing anomaly detection.

[0053] In one approach, for the collected performance index time-series data, missing and outlier values ​​in the collected performance index time-series data can be corrected first, and then features can be extracted from the performance index time-series data. Based on the extracted features, the performance index time-series data can be spliced ​​to obtain spliced ​​data. The spliced ​​data can be divided into training and testing sets. An anomaly detection model can be trained based on the optimized objective function. Then, the anomaly detection model can be tested and evaluated to obtain evaluation criteria. Using the evaluation criteria and the anomaly detection model, anomaly detection can be performed on the data to be detected.

[0054] While this scheme can detect time-series data of performance indicators, it uses an unsupervised learning method, which is complex and its accuracy needs to be improved. In addition, when judging abnormal data, it uses a preset abnormality score threshold to identify the sample points to be detected as abnormal sample points and outputs the detection results. This method also has some problems with low effectiveness.

[0055] The technical solution of this application fully utilizes the excellent features of Convolutional Neural Networks (CNNs) in feature extraction from multi-dimensional data sequences and the characteristics of Long Short-Term Memory (LSTM) in capturing dependencies and making predictions in time series. A hybrid neural network is used to model and train performance indicator time series data, and an attention mechanism is added to improve the accuracy of the predicted data baseline of the hybrid neural network model. Furthermore, to address the unreliability of a single data dimension, multi-dimensional performance indicator time series data is introduced, and a multi-feature fusion method is adopted to improve the persuasiveness and credibility of the anomaly detection results. In addition, this application embodiment also utilizes the predicted data baseline, employing a moving window overlay of multiple peaks in the data baseline combined with a threshold mode to determine and detect anomalies. The technical solution of this application embodiment can guarantee the reliability and effectiveness of anomaly detection for performance indicator time series data.

[0056] Please see Figure 1 , Figure 1 This is a flowchart illustrating the data detection method provided in the embodiments of this application, as shown below. Figure 1 As shown, the data detection method provided in this application includes the following steps:

[0057] S101: Obtain the data to be detected, wherein the data to be detected is the time series data of the first performance index.

[0058] In this embodiment of the application, the time series data of performance indicators are generally some performance indicators and system status indicators during the operation of the network system. They are usually time series collected at fixed time intervals. There are some potential relationships and some specific patterns among these data.

[0059] Here, the first performance index time series data is the performance index time series data after preprocessing the original performance index data of the collected network system.

[0060] In an optional embodiment of this application, step 101 specifically includes the following steps:

[0061] 1.1) Use a script program to collect time-series data of the second performance indicator of the network system according to the set time granularity;

[0062] 1.2) Preprocess the time series data of the second performance index to obtain the data to be tested.

[0063] Here, the second performance indicator time series data refers to the raw performance indicator time series data of the network system collected. Specifically, an automated script program (such as the automated program SHELL) can be used to collect the second performance indicator time series data. When collecting the performance indicator time series data, the data is collected according to the set time granularity. For example, if the time granularity is one minute, the network system's performance indicator time series data of multiple dimensions can be collected once per minute. These multiple dimensions may include transaction per second (TPS), central processing unit (CPU) utilization, network traffic volume, and other dimensions.

[0064] In one implementation, the collected second performance index time-series data can be structured into a fixed and unified format and then stored in a database, such as in a time-series database INFLUXDB.

[0065] In an optional embodiment of this application, step 1.2) above includes the following steps:

[0066] The missing data in the time series data of the second performance index is filled in, and the time series data of the second performance index after filling in the missing data is standardized to obtain the data to be tested.

[0067] Since the collected raw performance index time-series data, i.e., the second performance index time-series data, often contains missing values ​​at certain points, data imputation can be used to fill in the missing data in the raw performance index time-series host. For example, a moving window can be used to fill in the missing data, and the average of the data from the five points surrounding the missing data point can be used as the fill value. In addition, it is necessary to standardize the second performance index data to eliminate the differences in amplitude between the performance index time-series data of different dimensions.

[0068] S102: Input the data to be detected into the Convolutional Neural Network (CNN) and the Long Short-Term Memory Network (LSTM) respectively to obtain the first feature data and the second feature data.

[0069] Since performance indicator time series data are generally correlated rather than independent, and since performance indicator time series data is a time series, time series analysis can be performed on the correlation between the corresponding time series observations of performance indicator time series data. This allows for the extraction of as much information as possible from the time series, thereby more accurately revealing certain implicit characteristics of the network system and its development and change patterns. Feature extraction refers to extracting parameters that can represent the characteristics of the network system from the time series.

[0070] In a network system, anomalies in a single data dimension are not necessarily caused entirely by the network system itself. Among the many data dimensions observed in a network system, there are some inherent relationships. If these inherently related data dimensions can be gathered together for analysis, the results of anomaly detection will be more convincing and more accurate.

[0071] The solution in this application embodiment can detect abnormal performance indicator time series data by using deep learning methods to analyze the inherent implicit features between data in multiple dimensions based on the multiple dimensions of performance indicator time series data in the network system.

[0072] This application employs a hybrid CNN and LSTM network approach for feature extraction from time-series performance indicator data. On one hand, it fully leverages CNN's convolution and pooling operations to extract implicit features between time-series performance indicator data, enabling better extraction of local features and the abstraction of these local features into high-level features. On the other hand, it combines LSTM, utilizing its time-expansion properties and long-term memory capabilities, to effectively predict time-series performance indicator data. Thus, when detecting outliers in time-series performance indicator data, the CNN+LSTM hybrid neural network model not only considers the spatial relationships between different dimensions of the time-series performance indicator data but also takes into account the temporal variations in the data.

[0073] In an optional embodiment of this application, the data to be detected includes data in n dimensions, where n is an integer greater than or equal to 1; step 102 specifically includes the following steps:

[0074] 2.1) Input the data to be detected into a CNN to obtain the features of the data to be detected in n dimensions; based on the features of the data to be detected in the n dimensions, obtain the first feature data; the first feature data is an m1*n dimensional vector;

[0075] 2.2) Input the data to be detected into an LSTM to obtain the features output by the data to be detected in the n hidden layers of the LSTM; the dimension of each of the n hidden layers is m2; based on the features output by the n hidden layers, the second feature data is obtained; the second feature data is an n*m2 dimensional vector.

[0076] Please see Figure 2 , Figure 2 A schematic diagram of the CNN and LSTM feature extraction process provided in the embodiments of this application. Figure 2 In the image, on the left is CNN extracting spatial features from time-series performance metrics, and on the right is LSTM expressing temporal features from time-series performance metrics. Figure 2In this context, Pi represents the i-th dimension of the performance metric time series data (e.g., TPS, CPU utilization, network bandwidth, etc.), T represents the time length of the performance metric time series data, Ci is the i-th feature extracted by the CNN, and hi is the output of the i-th hidden layer of the LSTM.

[0077] For example, m1 can take the value 1. In this case, the CNN model can extract a 1*n dimensional feature vector C = [C1, C2, ..., Cn]. In the LSTM model, the input is time series data of performance index with length n, and the output dimension of the LSTM hidden layer is m2, resulting in an n*m2 dimensional feature vector.

[0078] S103: Based on the first feature data and the second feature data, obtain the data baseline of the data to be detected.

[0079] In this embodiment, a feature fusion mechanism is mainly used to fuse the first feature data and the second feature data to obtain the fused features of the performance index time series data.

[0080] In an optional embodiment of this application, step 103 specifically includes the following steps:

[0081] 3.1) The first feature data and the second feature data are fused using an attention mechanism to obtain a first weight;

[0082] 3.2) The first weight is normalized using a normalization function to obtain the second weight;

[0083] 3.3) Multiply the output of the hidden layer at each time point in the LSTM with the corresponding second weight in the second weight to obtain the third feature data;

[0084] 3.4) Predict the data baseline of the data to be detected based on the third feature data.

[0085] Figure 3 This diagram illustrates the feature fusion process using an attention mechanism, as provided in an embodiment of this application. In this embodiment, the attention mechanism allows the model to focus on the necessary portion of the data sequence input to the attention mechanism. Introducing the attention mechanism improves the scalability of the hybrid neural network model and the accuracy of the predicted data baseline.

[0086] The following formulas (1) to (3) illustrate the specific method of fusing the first feature data and the second feature data using the attention mechanism:

[0087] Φ(h i C) = tanh(h i*W a *C T +b a (1)

[0088]

[0089]

[0090] In the above formula, W a Given an m²*n dimensional predefined weight matrix, b a As a preset bias term, the features extracted by the CNN and the features extracted by the LSTM are fused into a set of weights Φ(h). i ,C), and for the obtained set of weights Φ(h) i C) α is obtained by normalization using the softmax function. i Then, the hidden layer output h at all time points of the LSTM is... i With the corresponding α i Multiply and then add them together to get the desired final characteristic expression f. map And use this feature to predict the data baseline of the time series data of performance indicators.

[0091] S104: Use the data baseline to determine the abnormal data corresponding to the data to be detected.

[0092] Traditional fixed-threshold early warning methods detect all performance indicator time-series data exceeding a fixed threshold as outliers. This crude approach misses many non-outlier data points within the performance indicator time-series data. This application's embodiment introduces a multi-peak moving window based on the data baseline to determine outlier data within the performance indicator time-series data, building upon the established threshold rules.

[0093] In an optional embodiment of this application, step 104 specifically includes the following steps:

[0094] 4.1) Identify m3 peaks that are higher than a set threshold among the multiple peaks included in the data baseline;

[0095] 4.2) Determine m3 detection windows based on the times corresponding to the m3 peak values;

[0096] 4.3) Determine the overlapping area of ​​the m3 detection windows, and identify the second performance index time series data whose value is greater than the maximum value among the m3 peaks in the overlapping area as abnormal data.

[0097] Specifically, in the process of detecting anomalies in performance indicator time-series data, in addition to using threshold rules for anomaly detection, the peak moving window of the predicted data baseline (S103) is used to further optimize the anomaly detection method. Since the predicted data baseline integrates historical performance data features and business characteristics, the pre- and post-peak coverage effect of the baseline peak is utilized when detecting performance indicator time-series data. Specifically, a predetermined time interval (e.g., 30 minutes before and after) before and after the peak time point in the predicted data baseline is used as the detection window. By combining multiple baseline peak moving windows with a comprehensive judgment method that compares and sets thresholds, false positives for non-anomaly data in the performance indicator time-series data are reduced, improving the effectiveness of anomaly detection.

[0098] In an optional embodiment of this application, step 4.2) above includes the following steps:

[0099] For each of the m3 peaks, the region corresponding to the target time period corresponding to that time is determined as the detection window corresponding to that peak; the target time period is the time period included before and after that time period, which is a predetermined duration.

[0100] For example, using a CNN+LSTM hybrid model to output the predicted data baseline, assuming the baseline peaks at 16:08 and 16:54 are higher than a threshold, with the peak at 16:08 being H1 and the peak at 16:54 being H2, then the first peak baseline window H1 covers the region from 15:38 to 16:38 (30 minutes before and after), and the second peak baseline window H2 covers the region from 16:24 to 17:24. Under the influence of these two peak baseline windows, abnormal data intervals are detected. Specifically, data within the overlapping areas of the detection windows, where the actual original performance index time-series data value is greater than MAX(H1, H2), is considered abnormal data. If multiple baseline peaks exceed the set threshold, then the two nearest peaks are selected sequentially for abnormal data identification.

[0101] Please see Figure 4 , Figure 4 A schematic diagram illustrating the implementation of the data detection method provided in this application embodiment;

[0102] Figure 4 The data detection method shown includes the following steps:

[0103] S401: Performance indicator time series data acquisition.

[0104] Automated scripts (such as the Shell program) can be used to collect time-series performance metrics data. Specifically, when collecting time-series performance metrics data, data is collected according to the set time granularity. For example, if the time granularity is one minute, time-series performance metrics data of the network system in multiple dimensions can be collected once per minute. These multiple dimensions can include transactions per second (TPS), central processing unit (CPU) utilization, network traffic volume, and other dimensions.

[0105] S402: Preprocessing of time series data for performance indicators.

[0106] Since the collected performance indicator time-series data often contains missing values ​​at certain points, data imputation can be used to fill in the missing values. Additionally, the performance indicator time-series data needs to be standardized to eliminate differences in amplitude between different dimensions of the performance indicator time-series data.

[0107] S403: Training of the abnormal data detection model.

[0108] This step primarily employs a hybrid CNN and LSTM network for feature extraction from performance indicator time-series data. On one hand, it fully leverages CNN's convolution and pooling operations to extract implicit features between performance indicator time-series data, enabling better extraction of local features and the abstraction of these local features into high-level features. On the other hand, it combines LSTM, utilizing its time-scaling properties and long-term memory capabilities, to effectively predict performance indicator time-series data. Thus, when detecting outliers in performance indicator time-series sequences, the CNN+LSTM hybrid neural network model not only considers the spatial relationships between different dimensions of performance indicator time-series data but also takes into account the temporal variations in the performance indicator time-series data.

[0109] Subsequently, an attention mechanism is employed to focus the model on the necessary portion of the data sequence within the input attention mechanism. The model incorporating the attention mechanism improves the accuracy of baseline prediction.

[0110] S404: Performance indicator timing data anomaly detection.

[0111] In the process of detecting anomalies in performance indicator time-series data, based on the use of set threshold rules for anomaly detection, the data baseline predicted by S403 is used, and the peak moving window of the predicted data baseline is further optimized for anomaly detection. Since the predicted data baseline integrates historical performance data features and business characteristics, the pre- and post-peak coverage effect of the baseline is utilized when detecting performance indicator time-series data. Specifically, a predetermined time interval (e.g., 30 minutes before and after) before and after the peak time point in the predicted data baseline is used as the detection window. By combining multiple baseline peak moving windows with a comprehensive judgment method that compares and sets thresholds, false positives for non-anomaly data in performance indicator time-series data can be reduced, improving the effectiveness of anomaly detection.

[0112] The technical solution of this application embodiment adopts a method that combines two neural networks, CNN and LSTM, and incorporates an attention mechanism to construct an anomaly detection model. The resulting anomaly detection model has both temporal and spatial feature representation capabilities, thereby improving the accuracy of anomaly detection.

[0113] Please see Figure 5 , Figure 5 This is a schematic diagram of the structural composition of the data detection device provided in the embodiments of this application, as shown below. Figure 5 As shown, the data detection device provided in this application embodiment includes:

[0114] The acquisition unit 501 is used to acquire the data to be detected, wherein the data to be detected is the time series data of the first performance index;

[0115] The first processing unit 502 is used to input the data to be detected into the convolutional neural network (CNN) and the long short-term memory network (LSTM) respectively to obtain the first feature data and the second feature data.

[0116] The second processing unit 503 is used to obtain the data baseline of the data to be detected based on the first feature data and the second feature data;

[0117] The determination unit 504 is used to determine abnormal data corresponding to the data to be detected using the data baseline.

[0118] In an optional embodiment of this application, the obtaining unit 501 is specifically used to: collect time-series data of the second performance index of the network system according to a set time granularity using a script program; and preprocess the time-series data of the second performance index to obtain data to be detected.

[0119] In an optional embodiment of this application, the obtaining unit 501 is further specifically used to: fill in the missing data in the second performance index time series data, and perform standardization processing on the second performance index time series data after filling in the missing data to obtain the data to be detected.

[0120] In an optional embodiment of this application, the data to be detected includes data in n dimensions, where n is an integer greater than or equal to 1; the first processing unit 502 is specifically used for: inputting the data to be detected into a CNN to obtain features of the data to be detected in n dimensions; obtaining first feature data based on the features of the data to be detected in the n dimensions; the first feature data is an m1*n dimensional vector; inputting the data to be detected into an LSTM to obtain features output by the data to be detected in the n hidden layers of the LSTM; the dimension of each of the n hidden layers is m2; obtaining second feature data based on the features output by the n hidden layers; the second feature data is an n*m2 dimensional vector.

[0121] In an optional embodiment of this application, the second processing unit 503 is specifically used for: fusing the first feature data and the second feature data using an attention mechanism to obtain a first weight; normalizing the first weight using a normalization function to obtain a second weight; multiplying the output of the hidden layer at each time point included in the LSTM with the corresponding second weight in the second weight to obtain third feature data; and predicting the data baseline of the data to be detected based on the third feature data.

[0122] In an optional embodiment of this application, the determining unit 504 is specifically used to: determine m3 peaks that are higher than a set threshold among the multiple peaks included in the data baseline; determine m3 detection windows based on the time corresponding to the m3 peaks; determine the overlapping area of ​​the m3 detection windows, and determine the second performance index time series data whose value is greater than the maximum value among the m3 peaks in the second performance index time series data corresponding to the overlapping area as abnormal data.

[0123] In an optional embodiment of this application, the determining unit 504 is further specifically configured to: for each of the m3 peaks, determine the region corresponding to the target time period corresponding to that time period as the detection window corresponding to that peak period; the target time period is the time period included by a predetermined duration before and after that time period.

[0124] Those skilled in the art should understand that Figure 5 The functions of each unit in the data detection device shown can be understood by referring to the relevant description of the aforementioned data detection method. Figure 5The functions of each unit in the data detection device shown can be implemented by a program running on a processor or by specific logic circuits.

[0125] This application also provides an electronic device. Figure 6 This is a schematic diagram of the hardware structure of the electronic device according to an embodiment of this application, such as... Figure 6 As shown, the electronic device includes: a communication component 603 for data transmission, at least one processor 601, and a memory 602 for storing computer programs capable of running on the processor 601. The various components in the terminal are coupled together via a bus system 604. It is understood that the bus system 604 is used to implement communication between these components. In addition to a data bus, the bus system 604 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 6 The general designated all buses as Bus System 604.

[0126] Wherein, when the processor 601 executes the computer program, it performs at least the following: Figure 1 The steps of the method shown.

[0127] It is understood that memory 602 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 602 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0128] The methods disclosed in the embodiments of this application can be applied to or implemented by processor 601. Processor 601 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuit of the hardware in processor 601 or by instructions in software form. The processor 601 may be a general-purpose processor, DSP, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 601 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in memory 602. Processor 601 reads the information in memory 602 and, in conjunction with its hardware, completes the steps of the aforementioned method.

[0129] In an exemplary embodiment, the electronic device may be implemented by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general-purpose processors, controllers, MCUs, microprocessors, or other electronic components to perform the aforementioned call recording method.

[0130] This application also provides a computer-readable storage medium storing a computer program thereon, characterized in that the program, when executed by a processor, is at least used to perform... Figure 1 The steps of the method are shown. The computer-readable storage medium may specifically be a memory. The memory may be, for example... Figure 6 The memory 602 shown.

[0131] The technical solutions described in the embodiments of this application can be combined arbitrarily without conflict.

[0132] In the several embodiments provided in this application, it should be understood that the disclosed methods and smart devices can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0133] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0134] In addition, each functional unit in the various embodiments of this application can be integrated into a second processing unit, or each unit can be a separate unit, or two or more units can be integrated into a unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0135] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data detection method, characterized in that, The method includes: The script program collects time-series data of the second performance indicator of the network system according to the set time granularity. A moving window is used to fill in the missing data in the second performance index time series data. The average of the data of multiple points around the missing data point is taken as the filling value. The second performance index time series data after missing data filling is standardized to obtain the data to be detected. The data to be detected is the first performance index time series data. The standardization process is used to eliminate the difference in amplitude of the performance index time series data of different dimensions. The data to be detected is input into a convolutional neural network (CNN) and a long short-term memory network (LSTM) to obtain the first feature data and the second feature data, respectively. Based on the first feature data and the second feature data, the data baseline of the data to be detected is obtained; The data baseline includes multiple peaks, among which m3 peaks are higher than a set threshold, where m3 is an integer greater than or equal to 2; For each of the m3 peaks, the region corresponding to the target time period corresponding to that time is determined as the detection window corresponding to that peak; the target time period is the time period included before and after that time period of a predetermined duration; The overlapping area of ​​m3 detection windows is determined, and the second performance index time series data with values ​​greater than the maximum value among the m3 peaks corresponding to the overlapping area is identified as abnormal data.

2. The method according to claim 1, characterized in that, The data to be detected includes data in n dimensions, where n is an integer greater than or equal to 1; the step of inputting the data to be detected into a convolutional neural network (CNN) and a long short-term memory network (LSTM) to obtain first feature data and second feature data includes: The data to be detected is input into a CNN to obtain the features of the data to be detected in n dimensions; based on the features of the data to be detected in the n dimensions, the first feature data is obtained; the first feature data is an m1*n dimensional vector, where m1 is an integer greater than or equal to 1; The data to be detected is input into an LSTM to obtain the features output by the hidden layer of the LSTM; the dimension of the hidden layer is m2, where m2 is an integer greater than or equal to 1; based on the features output by the hidden layer, second feature data is obtained; the second feature data is an n*m2 dimensional vector.

3. The method according to claim 2, characterized in that, The process of obtaining the data baseline of the data to be detected based on the first feature data and the second feature data includes: The first feature data and the second feature data are fused using an attention mechanism to obtain a first weight; The first weight is normalized using a normalization function to obtain the second weight; The output of the hidden layer at each time point in the LSTM is multiplied by the second weight corresponding to each time point in the second weight to obtain the third feature data. The data baseline of the data to be detected is predicted based on the third feature data.

4. A data detection device, characterized in that, The device includes: The acquisition unit is used to collect time-series data of the second performance index of the network system according to a set time granularity using a script program; it fills in the missing data in the second performance index time-series data by using a moving window, takes the average of the data of multiple points around the missing data point as the filling value, and performs standardization processing on the second performance index time-series data after missing data filling to obtain the data to be detected, wherein the data to be detected is the first performance index time-series data, and the standardization processing is used to eliminate the difference in amplitude of the performance index time-series data of different dimensions. The first processing unit is used to input the data to be detected into the convolutional neural network (CNN) and the long short-term memory network (LSTM) respectively to obtain the first feature data and the second feature data. The second processing unit is used to obtain the data baseline of the data to be detected based on the first feature data and the second feature data; The determining unit is configured to determine m3 peaks that are higher than a set threshold among the multiple peaks included in the data baseline, where m3 is an integer greater than or equal to 2; for each of the m3 peaks, the region corresponding to the target time period corresponding to that time is determined as the detection window corresponding to that peak; the target time period is the time period included before and after the predetermined duration; the overlapping region of the m3 detection windows is determined, and the second performance index time series data whose value is greater than the maximum value among the m3 peaks in the second performance index time series data corresponding to the overlapping region is determined as abnormal data.

5. An electronic device, characterized in that, The electronic device includes a memory and a processor, wherein the memory stores computer-executable instructions, and the processor, when executing the computer-executable instructions in the memory, can implement the method of any one of claims 1 to 3.

6. A computer storage medium, characterized in that, The storage medium stores executable instructions that, when executed by a processor, implement the method of any one of claims 1 to 3.

Images