Data processing method, device and computer readable storage medium

Abstract

The application discloses a data processing method, device and computer readable storage medium. The data processing method comprises the following steps: obtaining a target data sequence; obtaining a first abnormal data segment in the target data sequence; obtaining a first data search space in the target data sequence; obtaining a second abnormal data segment corresponding to the first abnormal data segment in the first data search space according to the first abnormal data segment; and labeling the second abnormal data segment. In the embodiment of the application, the first abnormal data segment and the first data search space are obtained in the target data sequence, so that the corresponding second abnormal data segment can be obtained and labeled in the first data search space according to the abnormal data segment template, the purpose of labeling other abnormal data segments in the target data sequence is achieved, and therefore, the labeling efficiency of abnormal data in the data can be improved, so that human resources and time resources can be saved.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of information processing technology, and particularly to a data processing method, apparatus, and computer-readable storage medium. Background Technology

[0002] With the development of big data and artificial intelligence technologies, more intelligent and efficient machine learning techniques are increasingly being introduced into the operation and maintenance of communication networks, such as anomaly detection, trend prediction, and root cause analysis. These technologies typically rely on high-quality training datasets to achieve good application results, and reliable labeled data is an integral part of these high-quality training datasets. Furthermore, the tremendous success of deep learning technology in image and speech recognition in recent years is inseparable from the extensive manual annotation of labeled datasets.

[0003] However, manually labeling massive training datasets is extremely costly, requiring significant human and time resources. For example, a medium-sized network contains millions of time-series data points. Manually labeling all outliers in this dataset is an impossible task. Even using empirical formulas or other methods to assist labeling results in inaccurate and incomplete information. Therefore, improving the efficiency of outlier labeling is a pressing technical problem that needs to be solved. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] In a first aspect, embodiments of the present invention provide a data processing method, apparatus, and computer-readable storage medium that can improve the efficiency of labeling abnormal data in data.

[0006] Secondly, embodiments of the present invention provide a data processing method, including,

[0007] Obtain the target data sequence;

[0008] Obtain the first anomalous data segment in the target data sequence;

[0009] Obtain a first data search space in the target data sequence;

[0010] Based on the first abnormal data segment, obtain the second abnormal data segment corresponding to the first abnormal data segment in the first data search space;

[0011] The second abnormal data segment is labeled.

[0012] Thirdly, embodiments of the present invention also provide an apparatus, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the data processing method of the second aspect as described above.

[0013] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer-executable instructions for performing the data processing method described above.

[0014] This invention includes: acquiring a target data sequence; acquiring a first abnormal data segment in the target data sequence; acquiring a first data search space in the target data sequence; acquiring a second abnormal data segment corresponding to the first abnormal data segment in the first data search space; and labeling the second abnormal data segment. According to the solution provided by this invention, by acquiring a first abnormal data segment and a first data search space in the target data sequence, the first abnormal data segment can serve as an abnormal data segment template. This allows for the acquisition and labeling of the corresponding second abnormal data segment in the first data search space based on the template, thus achieving the purpose of labeling other abnormal data segments in the target data sequence. Therefore, compared to traditional manual labeling of abnormal data segments, the solution provided by this invention can improve the labeling efficiency of abnormal data in the data, thereby saving human and time resources.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in the description, claims, and drawings. Attached Figure Description

[0016] The accompanying drawings are provided to further understand the technical solutions of the present invention and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the technical solutions of the present invention, and do not constitute a limitation on the technical solutions of the present invention.

[0017] Figure 1 This is a schematic diagram of a system architecture platform for performing a data processing method according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of a data processing method provided in one embodiment of the present invention;

[0019] Figure 3 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0020] Figure 4This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0021] Figure 5 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0022] Figure 6 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0023] Figure 7 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0024] Figure 8 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0025] Figure 9 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0026] Figure 10 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0027] Figure 11 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0028] Figure 12 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0029] Figure 13 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0030] Figure 14 This is a flowchart of a data processing method provided in another embodiment of the present invention;

[0031] Figure 15 This is a flowchart of a heuristic algorithm provided in one embodiment of the present invention;

[0032] Figure 16 This is a flowchart of a heuristic algorithm provided in another embodiment of the present invention;

[0033] Figure 17 This is a main flowchart of a data processing method provided in another embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0035] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, or the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0036] For data collected during network operation, such as time-series metrics, most time-series metrics exhibit periodic characteristics. Many recurring anomalies often appear at the same location across different periods, and their patterns tend to be similar. Specifically, in a long-lasting time-series metric with a large amount of anomalies, most anomalies can be categorized into a few types with similar characteristics; truly unique anomalies are relatively few. Furthermore, similar anomalies can also exist between closely related time-series metrics. For example, if a network element experiences an abnormally high CPU utilization rate during a certain period, this situation may also appear in the CPU utilization time-series data of another network element handling similar services.

[0037] Based on the above, the present invention provides a data processing method, device, and computer-readable storage medium. According to the periodic characteristics of recurring anomalous data in most data, by obtaining a first anomalous data segment and a first data search space in the target data sequence, the first anomalous data segment can serve as an anomalous data segment template. Therefore, a corresponding second anomalous data segment can be obtained and labeled in the first data search space based on the template. This achieves the purpose of labeling other anomalous data segments in the target data sequence. Thus, for time series index data with a large amount of anomalous data but few anomalous types, compared to traditional manual labeling of anomalous data segments, the solution provided by the embodiments of the present invention can improve the labeling efficiency of anomalous data in the data, thereby saving human and time resources.

[0038] The embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0039] like Figure 1 As shown, Figure 1 This is a schematic diagram of a system architecture platform for performing a data processing method provided in an embodiment of the present invention.

[0040] exist Figure 1In this example, the system architecture platform includes a memory 110 and a processor 120, wherein the memory 110 and the processor 120 can be connected via a bus or other means. Figure 1 Taking the example of a connection between China and Israel via a bus.

[0041] Memory 110, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory 110 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 110 may optionally include memory remotely located relative to processor 120, and these remote memories can be connected to the system architecture platform via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0042] Those skilled in the art will understand that this system architecture platform can be applied to various network controllers or network managers, and this embodiment does not specifically limit it. Furthermore, a network controller or network manager with this system architecture platform can be applied to various network systems, such as 3G communication network systems, LTE communication network systems, 5G communication network systems, and subsequent evolved mobile communication network systems, and this embodiment does not specifically limit it.

[0043] It will be understood by those skilled in the art that Figure 1 The system architecture platform shown does not constitute a limitation on the embodiments of the present invention, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0044] exist Figure 1 In the system architecture platform shown, the processor 120 can call the data processing program stored in the memory 110 to execute the data processing method.

[0045] Based on the above system architecture platform, various embodiments of the data processing method of the present invention are proposed below.

[0046] like Figure 2 As shown, Figure 2 This is a flowchart of a data processing method provided in an embodiment of the present invention. The data processing method includes, but is not limited to, steps S100, S200, S300, S400 and S500.

[0047] Step S100: Obtain the target data sequence.

[0048] In one embodiment, the target data sequence can be time-series indicator data or other sequence data. Other sequence data can be non-time-series indicator data such as business type sequence data or business quantity sequence data; this embodiment does not impose specific limitations. Furthermore, the target data sequence can be automatically obtained from the network by a device equipped with the aforementioned system architecture platform, or it can be manually entered into the device equipped with the aforementioned system architecture platform; this embodiment does not impose specific limitations.

[0049] Step S200: Obtain the first abnormal data segment in the target data sequence.

[0050] In one embodiment, the first anomalous data segment is a data segment in the target data sequence containing anomalous data. This segment can be manually identified and selected, then entered into a device with the aforementioned system architecture platform so that the device can access the first anomalous data segment, or saved to the device's memory so that the device can retrieve it from the memory. Once the first anomalous data segment in the target data sequence is obtained, it provides the necessary foundation for subsequent steps of labeling the remaining anomalous data segments in the target data sequence.

[0051] It is worth noting that in time series index data, outliers often occur at one or more consecutive points in time. These points are called outlier points, and the data segment corresponding to the set of outlier points is called an outlier segment. An outlier segment may last for a relatively long time (i.e., contain many outlier points). Therefore, an outlier segment must at least have the following characteristics: a start point, an end point, at least three points, and no overlapping points between outliers.

[0052] Step S300: Obtain the first data search space in the target data sequence.

[0053] In one embodiment, the data search space is a subset of candidate anomalous data extracted from the target data sequence using machine learning methods. By acquiring the data search space, most of the normal data can be filtered out, preventing similar data segments appearing in the normal data from being mistakenly identified as similar anomalous data and thus improving the accuracy of searching for similar anomalous data. Furthermore, acquiring the data search space can also narrow down the search range for similar anomalous data, thereby improving search efficiency.

[0054] In one embodiment, the first abnormal data segment can be a data segment outside the first data search space or a data segment within the first data search space; this embodiment does not specifically limit this. When the first abnormal data segment is a data segment within the first data search space, since the abnormal data has already been initially extracted when obtaining the first data search space, obtaining the first abnormal data segment within the first data search space makes the acquisition of the first abnormal data segment more accurate and effective.

[0055] Step S400: Obtain the second abnormal data segment corresponding to the first abnormal data segment in the first data search space based on the first abnormal data segment.

[0056] In one embodiment, a first anomalous data segment can be used as an anomalous data segment template. Data segments in a first data search space are compared with this first anomalous data segment to identify data segments in the first data search space that are identical or similar to the first anomalous data segment. These identical or similar data segments are the second anomalous data segments. Therefore, by using the first anomalous data segment as a template to obtain the corresponding second anomalous data segment in the first data search space, the goal of finding anomalous data in the target data sequence can be achieved. Thus, for time series index data with a large amount of anomalous data but few types of anomalous data, this embodiment can improve the efficiency of finding anomalous data compared to traditional manual methods, thereby saving human and time resources.

[0057] In one embodiment, all data in the first data search space can be considered as a single data segment, and a similarity calculation can be performed using the Dynamic Time Warping (DTW) algorithm. When the calculation result is similar, a second anomalous data segment corresponding to the first anomalous data segment in the first data search space can be determined. Alternatively, the data in the first data search space can be divided into multiple data segments of equal length to the first anomalous data segment, and similarity calculations can be performed on the first anomalous data segment and the multiple data segments in the first data search space using methods such as Euclidean distance or Pearson correlation coefficient, thereby determining the second anomalous data segment corresponding to the first anomalous data segment in the first data search space.

[0058] Step S500: Mark the second abnormal data segment.

[0059] In one embodiment, once a second abnormal data segment corresponding to the first abnormal data segment is found in the first data search space, the second abnormal data segment can be labeled to facilitate the formation of abnormal label data, thereby obtaining a high-quality training dataset that can be used for machine learning techniques such as deep learning.

[0060] In one embodiment, since the data processing method uses the above-described steps S100, S200, S300, S400, and S500, by obtaining a first abnormal data segment and a first data search space in the target data sequence, the first abnormal data segment can be used as an abnormal data segment template. Therefore, a corresponding second abnormal data segment can be obtained and labeled in the first data search space based on the abnormal data segment template, thus achieving the purpose of labeling other abnormal data segments in the target data sequence. Therefore, for time series index data with a large amount of abnormal data but few types of abnormalities, compared to traditional manual labeling of abnormal data segments, the data processing method of this embodiment can improve the labeling efficiency of abnormal data in the data, thereby saving human and time resources.

[0061] Additionally, refer to Figure 3 In one embodiment, step S300 includes, but is not limited to, the following steps:

[0062] Step S310: Obtain the first abnormal feature value of the target data sequence;

[0063] Step S320: Determine the first data position in the target data sequence corresponding to the first abnormal feature value based on the first abnormal feature value;

[0064] Step S330: Obtain the first data search space based on the first data position.

[0065] Those skilled in the art will understand that anomalous data is data that deviates from most of the data in a dataset. Based on this, the first anomalous feature value in this embodiment refers to the deviation value between anomalous data and normal data.

[0066] In one embodiment, after obtaining the first abnormal feature value of the target data sequence, the first data position in the target data sequence corresponding to the first abnormal feature value can be determined based on the first abnormal feature value. That is, the position of abnormal data in the target data sequence can be determined based on the first abnormal feature value. For example, when a data deviates from most of the data by a distance greater than or equal to the first abnormal feature value, the position of the data can be determined as the location of an abnormal data.

[0067] In one embodiment, the first data location may include a starting abnormal location, an intermediate abnormal location, and an ending abnormal location. Once the starting abnormal location, the intermediate abnormal location, and the ending abnormal location are determined, the first data search space can be obtained.

[0068] It is worth noting that after obtaining the first outlier feature value, different algorithms such as LOF (Local Outlier Factor), DBSCAN, or Isolation Forest (iForest) can be used to obtain the first data search space. This embodiment does not impose any specific limitations. Taking the Isolation Forest algorithm as an example, firstly, an iTree is constructed based on the target data sequence, and the data in the target data sequence is used as the sample data for this tree. Then, the sample data is binary-partitioned using the first outlier feature value, separating the sample data that meets the first outlier feature value from the sample data that does not meet the first outlier feature value, forming two datasets respectively. Then, the above process is repeated for these two datasets until the data cannot be further divided or the maximum height of the tree is reached. Therefore, the first data position in the target data sequence corresponding to the first outlier feature value can be obtained, and the first data search space can be determined based on the first data position.

[0069] Furthermore, those skilled in the art will understand that the LOF algorithm, DBSCAN algorithm, and Isolation Forest algorithm are all commonly used algorithms in this field. Therefore, the specific principles of these algorithms will not be elaborated here.

[0070] Additionally, refer to Figure 4 In one embodiment, step S310 includes, but is not limited to, the following steps:

[0071] Step S311: Obtain the first baseline prediction data of the target data sequence;

[0072] Step S312: Obtain the first abnormal feature value based on the deviation between the first baseline predicted data and the data in the target data sequence.

[0073] In one embodiment, when performing the step of obtaining the first abnormal feature value of the target data sequence, the baseline prediction data of the normal data in the target data sequence can be obtained first, and then the first abnormal feature value can be obtained based on the deviation (i.e., the absolute difference) between the data in the target data sequence and the baseline prediction data.

[0074] In one embodiment, different baseline prediction methods can be used to obtain baseline prediction data for normal data in the target data sequence. This embodiment does not impose specific limitations. For example, the baseline prediction method can employ time-series baseline prediction methods such as differencing, moving average, weighted moving average, exponentially weighted moving average, differential moving average autoregressive method, or triple exponential smoothing. It can also employ regression methods such as random forest and XGBoost (Xtreme Gradient Boosting). By employing multiple baseline prediction methods, multiple first anomaly feature values ​​can be obtained. Furthermore, by combining different first anomaly feature values ​​and performing corresponding steps to obtain the first data search space, the accuracy and generalization ability of the first data search space can be improved.

[0075] In one embodiment, after obtaining the first baseline prediction data of the target data sequence, the first anomaly feature value can be obtained by the following formula:

[0076] R = |P i -X i |

[0077] Where R is the first abnormal feature value, X i For the data in the target data sequence, P i The first baseline prediction data for the target data sequence.

[0078] Those skilled in the art will understand that the difference method, moving average method, weighted moving average method, exponentially weighted moving average method, differential moving average autoregressive method, triple exponential smoothing method, random forest and XGBooste are all commonly used algorithms in this field. Therefore, the specific principles of these algorithms will not be elaborated here.

[0079] Additionally, refer to Figure 5 In one embodiment, step S400 includes, but is not limited to, the following steps:

[0080] Step S410: Determine the third data segment in the first data search space;

[0081] Step S420: Calculate the similarity between the first abnormal data segment and the third data segment to obtain the first similarity metric value corresponding to the third data segment;

[0082] Step S430: Determine the corresponding third data segment as the second abnormal data segment based on the first similarity metric value.

[0083] In one embodiment, when performing the step of obtaining the second abnormal data segment corresponding to the first abnormal data segment in the first data search space based on the first abnormal data segment, a third data segment can be determined first in the first data search space. After determining the third data segment, a similarity measurement algorithm is used to calculate the similarity between the first abnormal data segment and the third data segment to obtain a first similarity measurement value corresponding to the third data segment. When the first similarity measurement value indicates that the first abnormal data segment and the third data segment are similar, the corresponding third data segment can be determined as the second abnormal data segment (i.e., the remaining abnormal data segments in the first data search space). That is, by comparing the similarity between the first abnormal data segment and the third data segment, it is determined whether the third data segment is the second abnormal data segment. Compared with the traditional manual annotation of abnormal data segments, this embodiment can improve the annotation efficiency of abnormal data in the data, thereby saving human and time resources.

[0084] In one embodiment, the number of the third data segments can be one or more, and this embodiment does not make a specific limitation. When the number of the third data segments is one, all the data in the first data search space can be determined as the third data segment, or a portion of the continuous data in the first data search space can be determined as the third data segment, and this embodiment does not make a specific limitation. When the number of the third data segments is multiple, the data in the first data search space can be divided into multiple data segments of equal length, or the data in the first data search space can be divided into multiple data segments of unequal length, and this embodiment does not make a specific limitation.

[0085] In one embodiment, the similarity calculation between the first anomalous data segment and the third data segment can be implemented using different similarity measurement algorithms. For example, for multiple third data segments of equal length, Euclidean distance, Pearson correlation coefficient, or Spearman rank correlation coefficient can be used to calculate the similarity between the first anomalous data segment and the third data segment. Alternatively, for multiple third data segments of unequal length, the DTW algorithm or an improved fast DTW algorithm can be used to calculate the similarity between the first anomalous data segment and the third data segment. The specific implementation method for calculating the similarity between the first anomalous data segment and the third data segment can be appropriately selected according to actual needs, and this embodiment does not impose specific limitations. It is worth noting that the improved fast DTW algorithm may include the FastDTW algorithm, SparseDTW algorithm, LB_Keogh algorithm, and LB_Improved algorithm, etc. Among them, the FastDTW algorithm can reduce computational complexity by limiting and narrowing the search space and data abstraction methods, even when the accuracy difference is not significant.

[0086] Those skilled in the art will understand that Euclidean distance, Pearson correlation coefficient, Spearman rank correlation coefficient, DTW algorithm, and various improved fast DTW algorithms are all commonly used algorithms in this field. Therefore, the specific principles of these algorithms will not be elaborated here.

[0087] In another embodiment, step S430 includes, but is not limited to, the following steps:

[0088] Step S431: When the first similarity metric value is less than a preset threshold, the third data segment corresponding to the first similarity metric value is determined to be the second abnormal data segment.

[0089] In one embodiment, the first similarity metric value represents the degree of similarity between the first abnormal data segment and the third data segment. The smaller the value of the first similarity metric value, the higher the degree of similarity between the first abnormal data segment and the third data segment. Therefore, when the first similarity metric value is less than a preset threshold, it can be determined that the first abnormal data segment and the third data segment have a high degree of similarity. Thus, the third data segment corresponding to the first similarity metric value can be determined as the second abnormal data segment.

[0090] In one embodiment, the preset threshold can be appropriately selected according to the different similarity measurement algorithms used. For example, different preset thresholds can be used for Euclidean distance and DTW algorithms. This embodiment does not make specific limitations.

[0091] Additionally, refer to Figure 6 In one embodiment, when the number of third data segments is two or more, step S430 may include, but is not limited to, the following steps:

[0092] Step S432: Obtain the first similarity metric value that is less than a preset threshold;

[0093] Step S433: Sort the first similarity metric values ​​that are less than the preset threshold from smallest to largest to adjust the sorting of the corresponding third data segment;

[0094] Step S434: Determine the first N third data segments as the second abnormal data segments, where N is greater than or equal to 1.

[0095] In one embodiment, when there are two or more third data segments, the number of first similarity metrics corresponding to the third data segments is also two or more. In this case, first similarity metrics with values ​​less than a preset threshold can be obtained first to filter out third data segments that have a certain degree of similarity to the first abnormal data segments and exclude the remaining third data segments with lower similarity. Then, the first similarity metrics with values ​​less than the preset threshold are sorted from smallest to largest to adjust the sorting of the corresponding third data segments, so that the third data segments that have a certain degree of similarity to the first abnormal data segments can be reordered from high to low similarity. Then, according to the actual application, the first few third data segments are determined as the second abnormal data segments.

[0096] To illustrate with a specific example, the FastDTW algorithm can be used to compare each third data segment in the first data search space with the first anomalous data segment to calculate the similarity metric value of each third data segment. Then, all third data segments are sorted according to the similarity metric value to obtain several third data segments with a high degree of similarity to the first anomalous data segment. Based on these several third data segments, the second anomalous data segment that needs to be labeled can be determined.

[0097] In one embodiment, the optimal value of N may differ for different first anomalous data segments and different first data search spaces. For example, if the value of N is too small, some anomalous data segments may be missed and not labeled, while if the value of N is too large, precision may decrease due to the identification of some anomalous data segments with low similarity. Therefore, the value of N needs to be appropriately selected based on the actual application. To select the optimal value of N more accurately, the AUC value can be calculated by establishing a precision-recall curve.

[0098] It is worth noting that precision refers to the proportion of correctly labeled outlier data segments; recall refers to the proportion of manually labeled outlier data segments that are correctly labeled; AUC (Are Under Curve) is a model evaluation metric. Simply put, it involves randomly selecting a pair of samples (one positive and one negative), and then using the trained classifier to predict the probability of predicting the positive sample as greater than the probability of predicting the negative sample.

[0099] Additionally, refer to Figure 7 In one embodiment, step S100 may include, but is not limited to, the following steps:

[0100] Step S110: Obtain multiple data sequences to be tested;

[0101] Step S120: Cluster the multiple test data sequences to obtain the target data class;

[0102] Step S130: Determine a target data sequence from each target data class.

[0103] In one embodiment, the number of test data sequences collected from the network is enormous. Manually identifying the first anomalous data segment in each sequence would be extremely labor-intensive. Furthermore, there are cases where the data sequence was collected in a short time, resulting in a lack of similar anomalous data. Therefore, obtaining the first anomalous data segment in these sequences becomes difficult. However, for a single data metric, numerous test data sequences can be collected in the network based on the different resource objects it is bound to. For example, in a medium-sized network, there are tens of thousands of port resources. Taking port traffic as an example, tens of thousands of test data sequences can be collected, and these sequences often exhibit certain similarities. For instance, the time-series data used to statistically analyze the access port traffic of a base station deployed at school A and the time-series data used to statistically analyze the access port traffic of a base station deployed at school B will be quite similar due to the similar daily routines of students at schools A and B. Furthermore, the anomalous data in similar test data sequences will share certain common characteristics. Based on the above, we can first perform clustering on the multiple data sequences to be tested to obtain target data classes, and then determine a target data sequence from each target data class, thereby providing the necessary basic conditions for subsequent steps.

[0104] In one embodiment, the number of target data classes obtained by clustering multiple test data sequences can be one or more, depending on the similarity of the test data sequences. For example, if the multiple test data sequences are relatively similar, then all test data sequences can be classified into one target data class. If some test data sequences among the multiple test data sequences are relatively similar, then the multiple test data sequences can be divided into multiple target data classes, and each target data class includes a portion of the test data sequences.

[0105] Additionally, refer to Figure 8 In one embodiment, the data processing method may further include the following steps:

[0106] Step S600: Obtain the second data search space from the remaining test data sequences in each target data class;

[0107] Step S700: Using the first abnormal data segment in the target data sequence, obtain the second abnormal data segment in the second data search space of the remaining test data sequences.

[0108] In one embodiment, when a target data sequence is determined in each target data class, the second abnormal data segment in the target data sequence can be obtained and labeled through the steps in the above embodiments. The specific technical principles and the resulting technical effects can be referred to the relevant descriptions in the above embodiments, and will not be repeated here.

[0109] In one embodiment, since the test data sequences in each target data class have a certain similarity, the first abnormal data segment obtained in the target data sequence can be applied to the acquisition and annotation of the second abnormal data segment for the remaining test data sequences in the same target data class. Therefore, a second data search space can be obtained first in the remaining test data sequences in each target data class, and then the first abnormal data segment in the target data sequence can be used to obtain the second abnormal data segment in the second data search space of the remaining test data sequences. Since the second abnormal data segment can be obtained in the second data search space of the remaining test data sequences based solely on the first abnormal data segment in the target data sequence, the operation steps of obtaining the corresponding first abnormal data segment in each of the remaining test data sequences can be saved. This allows for a simpler and more efficient acquisition of the second abnormal data segment in each test data sequence, thereby improving the annotation efficiency of abnormal data in multiple time series index data.

[0110] It is worth noting that the second data search space in this embodiment is the same type of technical feature as the first data search space in the above embodiments. The only difference between the two is their respective objects of reference. The first data search space belongs to the target data sequence, while the second data search space belongs to the other test data sequences within the same target data class. To avoid repetition, the second data search space will not be described in detail here. For explanations regarding the second data search space, please refer to the explanations regarding the first data search space in the above embodiments.

[0111] Furthermore, it is worth noting that step S700 in this embodiment is the same as that described above. Figure 2Step S400 in the illustrated embodiment is similar, and both have similar technical principles and effects. The only difference lies in the execution object. In the above embodiment, the execution object of step S400 is the first data search space of the target data sequence, while in this embodiment, the execution object of step S700 is the second data search space of the remaining test data sequences in the same target data class. To avoid repetition, step S700 will not be described in detail here. For explanations of step S700, please refer to the explanations of step S400 in the above embodiments.

[0112] Additionally, refer to Figure 9 In one embodiment, step S600 includes, but is not limited to, the following steps:

[0113] Step S610: Obtain the second abnormal feature value of the remaining test data sequence in each target data class;

[0114] Step S620: Determine the second data position corresponding to the second abnormal feature value in the remaining test data sequences according to the second abnormal feature value;

[0115] Step S630: Obtain the second data search space of the remaining test data sequences according to the second data position.

[0116] In one embodiment, the second abnormal feature value and the second data position in this embodiment belong to the same type of technical feature as the first abnormal feature value and the first data position in the above embodiments. The only difference between the two is their respective objects of reference. The first abnormal feature value and the first data position both belong to the target data sequence, while the second abnormal feature value and the second data position both belong to other test data sequences within the same target data class. To avoid repetition, the second abnormal feature value and the second data position will not be described in detail here. For explanations regarding the second abnormal feature value and the second data position, please refer to the explanations regarding the first abnormal feature value and the first data position in the above embodiments.

[0117] In one embodiment, steps S610, S620, and S630 of this embodiment are the same as those described above. Figure 3Steps S310, S320, and S330 in the illustrated embodiment are similar, sharing similar technical principles and effects. The only difference lies in the execution object. In the above embodiment, steps S310, S320, and S330 execute on the target data sequence, while in this embodiment, steps S610, S620, and S630 execute on other test data sequences within the same target data class. To avoid repetition, steps S610, S620, and S630 will not be described in detail here. For explanations of steps S610, S620, and S630, please refer to the explanations of steps S310, S320, and S330 in the above embodiment.

[0118] Additionally, refer to Figure 10 In one embodiment, step S610 includes, but is not limited to, the following steps:

[0119] Step S611: Obtain the second baseline prediction data of the remaining test data sequences in each target data class;

[0120] Step S612: Obtain the second abnormal feature value of the remaining test data sequence based on the deviation value between the second baseline prediction data and the data in the remaining test data sequence.

[0121] In one embodiment, the second baseline prediction data in this embodiment belongs to the same type of technical feature as the first baseline prediction data in the above embodiments. The only difference between the two is their respective objects of reference. The first baseline prediction data belongs to the target data sequence, while the second baseline prediction data belongs to other test data sequences within the same target data class. To avoid repetition, the second baseline prediction data will not be described in detail here. For explanations regarding the second baseline prediction data, please refer to the explanations regarding the first baseline prediction data in the above embodiments.

[0122] In one embodiment, steps S611 and S612 in this embodiment are the same as those described above. Figure 4 Steps S311 and S312 in the illustrated embodiment are similar, possessing similar technical principles and effects. The only difference lies in the execution object. In the above embodiment, the execution object of steps S311 and S312 is the target data sequence, while in this embodiment, the execution objects of steps S611 and S612 are the remaining test data sequences within the same target data class. To avoid repetition, steps S611 and S612 will not be described in detail here. For explanations of steps S611 and S612, please refer to the explanations of steps S311 and S312 in the above embodiments.

[0123] Additionally, refer to Figure 11 In one embodiment, step S700 includes, but is not limited to, the following steps:

[0124] Step S710: Determine the fourth data segment in the second data search space of the remaining test data sequences respectively;

[0125] Step S720: Calculate the similarity between the first abnormal data segment in the target data sequence and the fourth data segment in the remaining test data sequences to obtain the second similarity metric value corresponding to the fourth data segment.

[0126] Step S730: Based on the second similarity metric, determine the corresponding fourth data segment in the remaining test data sequences as the second abnormal data segment in the remaining test data sequences.

[0127] In one embodiment, the fourth data segment and the second similarity metric in this embodiment belong to the same type of technical feature as the third data segment and the first similarity metric in the above embodiments. The only difference between them is their respective objects of association. The third data segment belongs to the first data search space of the target data sequence, and the first similarity metric corresponds to the third data segment. The fourth data segment belongs to the second data search space of the other test data sequences in the same target data class, and the second similarity metric corresponds to the fourth data segment. To avoid repetition, the fourth data segment and the second similarity metric will not be described in detail here. For explanations of the fourth data segment and the second similarity metric, please refer to the explanations of the third data segment and the first similarity metric in the above embodiments.

[0128] In one embodiment, steps S710, S720, and S730 of this embodiment are the same as those described above. Figure 5 Steps S410, S420, and S430 in the illustrated embodiment are similar, possessing similar technical principles and effects. The only difference lies in the execution object. In the above embodiment, steps S410, S420, and S430 execute on the first data search space of the target data sequence, while in this embodiment, steps S710, S720, and S730 execute on the second data search space of the remaining test data sequences within the same target data class. To avoid repetition, steps S710, S720, and S730 will not be described in detail here. For explanations of steps S710, S720, and S730, please refer to the explanations of steps S410, S420, and S430 in the above embodiment.

[0129] In another embodiment, step S730 includes, but is not limited to, the following steps:

[0130] Step S731: When the second similarity metric is less than a preset threshold, the corresponding fourth data segment in the remaining test data sequence is determined to be the second abnormal data segment in the remaining test data sequence.

[0131] In one embodiment, the second similarity metric value represents the degree of similarity between the first abnormal data segment and the fourth data segment. The smaller the value of the second similarity metric value, the higher the degree of similarity between the first abnormal data segment and the fourth data segment. Therefore, when the second similarity metric value is less than a preset threshold, it can be determined that the first abnormal data segment and the fourth data segment have a high degree of similarity. Thus, the fourth data segment corresponding to the second similarity metric value can be determined as the second abnormal data segment.

[0132] In one embodiment, the preset threshold can be appropriately selected according to the different similarity measurement algorithms used. For example, different preset thresholds can be used for Euclidean distance and DTW algorithms. This embodiment does not make specific limitations.

[0133] Additionally, refer to Figure 12 In one embodiment, when the number of fourth data segments is two or more, step S730 may include, but is not limited to, the following steps:

[0134] Step S732: Obtain the second similarity measure value corresponding to the values ​​of the other test data sequences that are less than a preset threshold;

[0135] Step S733: Sort the second similarity measure values ​​that are less than the preset threshold from smallest to largest, so as to adjust the sorting of the corresponding fourth data segments in the remaining test data sequences respectively;

[0136] Step S734: Identify the first N fourth data segments as the second abnormal data segments in the remaining test data sequences, where N is greater than or equal to 1.

[0137] In one embodiment, steps S732, S733, and S734 of this embodiment are the same as those described above. Figure 6Steps S432, S433, and S434 in the illustrated embodiment are similar, sharing similar technical principles and effects. The only difference lies in the execution object. In the above embodiment, the execution object of steps S432, S433, and S434 is the target data sequence, while in this embodiment, the execution objects of steps S732, S733, and S734 are other test data sequences within the same target data class. To avoid repetition, steps S732, S733, and S734 will not be described in detail here. For explanations of steps S732, S733, and S734, please refer to the explanations of steps S432, S433, and S434 in the above embodiment.

[0138] Additionally, refer to Figure 13 In one embodiment, step S120 may include, but is not limited to, the following steps:

[0139] Step S121: Perform data preprocessing on multiple test data sequences to obtain multiple first preprocessed data sequences;

[0140] Step S122: Baseline extraction processing is performed on multiple first preprocessed data sequences to obtain multiple second preprocessed data sequences;

[0141] Step S123: Cluster multiple second preprocessed data sequences according to similarity to obtain the target data class.

[0142] In one embodiment, when multiple test data sequences need to be clustered, they can first be preprocessed to obtain multiple first preprocessed data sequences. Then, baseline extraction is performed on each of these first preprocessed data sequences to obtain multiple second preprocessed data sequences. Next, these second preprocessed data sequences are clustered according to similarity to obtain the target data class. After clustering these multiple test data sequences to obtain the corresponding target data class, the anomaly labeling process for each test data sequence can be transformed into anomaly labeling process for each target data class. This reduces the complexity and processing time, thereby improving the labeling efficiency of anomaly data in the data.

[0143] In one embodiment, baseline extraction processing is performed on the first preprocessed data sequence to smooth out abnormal and noisy parts in the test data sequence, thereby improving the accuracy of similarity measurement between the test data sequences.

[0144] It is worth noting that the baseline extraction process in step S122 of this embodiment is similar to that described above. Figure 4The steps for obtaining baseline prediction data using the baseline prediction method in the illustrated embodiment have similar technical principles. For the explanation of the baseline extraction process in step S122 of this embodiment, please refer to the above. Figure 4 The relevant explanations and descriptions of obtaining baseline prediction data using the baseline prediction method in the illustrated embodiments will not be repeated here.

[0145] Additionally, refer to Figure 14 In one embodiment, step S121 may include, but is not limited to, the following steps:

[0146] Step S1211: Perform missing value imputation on multiple test data sequences to obtain multiple imputed data sequences;

[0147] Step S1212: Perform data standardization on the multiple filled data sequences to obtain multiple first preprocessed data sequences.

[0148] In one embodiment, the test data sequence collected from the network may have different degrees of missing values ​​due to various reasons. These missing values ​​not only cause the lengths of the test data sequences to be different, making it difficult to use some similarity measurement algorithms, but also affect the accuracy of baseline extraction processing. In order to solve these problems, this embodiment first fills in these missing values ​​to obtain a filled data sequence, and then performs data standardization processing on the filled data sequence to obtain a first preprocessed data sequence.

[0149] In one embodiment, linear interpolation can be used to fill in missing values. Linear interpolation smooths the waveform of the data sequence under test, thus facilitating baseline extraction. For example, for a time series index, the location of missing values ​​can be determined based on temporal continuity. After determining the location of the missing value, the specific value to be filled can be obtained based on the data before and after the location of the missing value. For example, the average of the data before and after the missing value can be used as the specific value to be filled. Those skilled in the art will understand that linear interpolation is a commonly used algorithm in the field; therefore, the specific principles of this algorithm will not be elaborated here.

[0150] In one embodiment, data standardization of the padded data sequence can transform and map the test data sequence to a specific interval, thereby helping to eliminate the dimensional differences between different test data sequences and enabling them to be compared for similarity. This embodiment can use the Z-Score method for data standardization, and its calculation formula is as follows:

[0151]

[0152] Where, x′i For the first preprocessed data sequence, x i The data sequence to be tested, Let σ be the mean of the data sequence to be tested, and σ be the standard deviation of the data sequence to be tested.

[0153] In another embodiment, the clustering of multiple second preprocessed data sequences based on similarity in step S123 may specifically include, but is not limited to, the following steps:

[0154] Step S1231: Cluster multiple second preprocessed data sequences according to similarity using the DBSCAN algorithm; wherein, the parameters of the DBSCAN algorithm include a distance function, a neighborhood number threshold, and a neighborhood distance threshold; the results of the DBSCAN algorithm include the number of categories and the anomaly ratio.

[0155] As will be understood by those skilled in the art, the DBSCAN algorithm is one of the commonly used clustering algorithms. The DBSCAN algorithm does not require pre-determining the number of cluster centers. Key parameters of the DBSCAN algorithm include the distance function, the neighborhood number threshold, and the neighborhood distance threshold. The results of the DBSCAN algorithm include the number of classes and the outlier ratio.

[0156] In one embodiment, the distance function can be the Euclidean distance function; the neighborhood number threshold can be set to 4; and the neighborhood distance threshold needs to be dynamically estimated based on the dataset, and this parameter has a significant impact on the clustering results.

[0157] Additionally, refer to Figure 15 In one embodiment, the neighborhood distance threshold can be obtained through a heuristic algorithm, wherein the heuristic algorithm includes, but is not limited to, the following steps:

[0158] Step S810: Calculate the pairwise similarity between multiple second preprocessed data sequences using a distance function to obtain similarity matrix data;

[0159] Step S820: Calculate the k-dist distance based on the similarity matrix data to obtain the k-dist sequence;

[0160] Step S830: Obtain the initial distance threshold parameter based on the k-dist sequence;

[0161] Step S840: Adjust the initial distance threshold parameter to obtain the neighborhood distance threshold.

[0162] In one embodiment, the k-dist distance refers to the distance between a data object and its k-th nearest neighbor. When determining a suitable neighborhood distance threshold, the similarity between multiple second preprocessed data sequences can be calculated pairwise using a distance function such as the Euclidean distance function to form a similarity matrix. Then, the k-dist distance is calculated based on this similarity matrix to obtain a k-dist sequence. Next, an initial distance threshold parameter is obtained based on this k-dist sequence. Finally, the initial distance threshold parameter is adjusted to obtain a suitable neighborhood distance threshold. Once a suitable neighborhood distance threshold is obtained through the above heuristic algorithm, it can be applied to cluster multiple second preprocessed data sequences by similarity using the DBSCAN algorithm in the above embodiment to obtain the target data class.

[0163] In one embodiment, before executing step S810, initial thresholds such as the maximum neighborhood distance threshold, minimum length threshold, slope threshold, and slope difference threshold can be set. After these initial thresholds are set, steps S810, S820, S830, and S840 are executed.

[0164] In one embodiment, when step S820 is executed, after calculating the k-dist distance of each k-dist point based on the similarity matrix data, the obtained k-dist distances can be sorted in ascending order, and k-dist points with a k-dist distance of 0 and k-dist points with a k-dist distance exceeding the maximum distance threshold of the neighborhood can be excluded. Therefore, the remaining k-dist points constitute the k-dist sequence.

[0165] In another embodiment, step S830 may include, but is not limited to, the following steps:

[0166] Step S831: Calculate the slope of each k-dist point in the k-dist sequence relative to its two adjacent points. If the slopes of the two adjacent points are both less than a preset slope threshold and the difference between the slopes of the two adjacent points is less than a preset slope difference threshold, then determine the current k-dist point as a candidate distance threshold.

[0167] Step S832: Determine the candidate distance threshold with the largest value as the initial distance threshold parameter.

[0168] In one embodiment, when it is necessary to perform the step of obtaining the initial distance threshold parameter based on the k-dist sequence, the relatively gentle k-dist points in the k-dist sequence can be determined as candidate distance thresholds first. The specific steps can be as follows: first calculate the slope of each k-dist point in the k-dist sequence with its two adjacent points before and after it. If the slope of the current k-dist point with its previous adjacent point (which can be defined as the left slope) and the slope of the current k-dist point with its next adjacent point (which can be defined as the right slope) are both less than a preset slope threshold, and the difference between the left slope and the right slope is less than a preset slope difference threshold, then the current k-dist point can be determined as a candidate distance threshold. When multiple candidate distance thresholds are obtained, these candidate distance thresholds can be sorted from largest to smallest, and then the candidate distance threshold with the largest value can be taken as the initial distance threshold parameter.

[0169] In another embodiment, step S840 may include, but is not limited to, the following steps:

[0170] Step S841, obtain the step length;

[0171] Step S842: Adjust the initial distance threshold parameter according to the step length to obtain the distance adjustment threshold. When the number of categories decreases, determine the distance adjustment threshold obtained in the previous step as the neighborhood distance threshold.

[0172] In one embodiment, after determining the initial distance threshold parameter, this initial distance threshold parameter can be further optimized. It is worth noting that when optimizing this initial distance threshold parameter, it is necessary to reduce the anomaly ratio as much as possible while keeping the number of categories constant. Since the anomaly ratio will continuously decrease as the value of the initial distance threshold parameter increases, while the number of categories may also decrease, a step-by-step approach can be used to gradually increase the value of the initial distance threshold parameter to determine the optimal neighborhood distance threshold. That is, based on the initial distance threshold parameter, the number of categories and the anomaly ratio are recalculated after each step increase until the number of categories decreases, at which point the step increase stops. At this point, the distance adjustment threshold obtained from the previous step can be determined as the optimal neighborhood distance threshold.

[0173] In one embodiment, the step length can be set based on an empirical value or based on a candidate distance threshold. For example, when the step length is set based on a candidate distance threshold, the step length can be set to one-tenth of the difference between the maximum distance threshold and the minimum distance threshold among the candidate distance thresholds. This embodiment does not specifically limit this.

[0174] To better illustrate the heuristic algorithm provided in the above embodiments, a detailed description is given below using specific examples:

[0175] In a specific example, such as Figure 16 As shown, the heuristic algorithm specifically includes the following steps:

[0176] Step S901, threshold setting.

[0177] In this step, initial thresholds are set, including the maximum neighborhood distance threshold, the minimum length threshold, the slope threshold, and the slope difference threshold.

[0178] Step S902: Calculate the sequence similarity matrix.

[0179] In this step, the similarity between each pair of data sequences is calculated pairwise using a distance function to form a similarity matrix.

[0180] Step S903: Calculate and sort the k-dist distances.

[0181] In this step, the k-dist distance of each k-dist point is calculated based on the similarity matrix data and sorted in ascending order.

[0182] Step S904: Filter by the maximum distance threshold.

[0183] In this step, k-dist points with a k-dist distance of 0 and k-dist points with a k-dist distance exceeding the maximum neighborhood distance threshold are excluded.

[0184] Step S905: Take the k-dist sequence values ​​in sequence.

[0185] Step S906: Determine whether both the left slope and the right slope are less than the slope threshold.

[0186] In this step, the slope of each k-dist point and its two adjacent points is calculated. If the slope of the current k-dist point with the previous adjacent point (which can be defined as the left slope) and the slope of the current k-dist point with the next adjacent point (which can be defined as the right slope) are both less than the preset slope threshold, then step S907 is executed; otherwise, step S905 is executed.

[0187] Step S907: Determine whether the difference between the left slope and the right slope is less than the slope difference threshold.

[0188] In this step, if the difference between the left slope and the right slope is less than the slope difference threshold, then step S908 is executed; otherwise, step S905 is executed.

[0189] Step S908: Determine the current k-dist point as a candidate distance threshold, and repeat steps S905 to S907. After obtaining all candidate distance thresholds, execute step S909.

[0190] Step S909: After sorting the candidate thresholds from largest to smallest, take the largest candidate threshold as the initial distance threshold parameter.

[0191] Step S910: Execute the clustering algorithm to obtain the number of categories and the anomaly ratio.

[0192] Step S911: Increase the step size based on the initial distance threshold parameter and execute the clustering algorithm.

[0193] Step S912: Determine if the number of categories has decreased. If yes, proceed to step S913; otherwise, proceed to step S911.

[0194] Step S913: Determine the previous distance threshold as the optimal distance threshold.

[0195] In another embodiment, step S130 may include, but is not limited to, the following steps:

[0196] Step S131: In each target data class, calculate the average distance between each test data sequence and the other test data sequences, and determine the test data sequence corresponding to the one with the smallest average distance as the target data sequence.

[0197] In one embodiment, after clustering multiple test data sequences to obtain target data classes, a core data sequence representing the corresponding target data class can be determined for each target data class; that is, a target data sequence is determined from each target data class. When determining a target data sequence from a target data class, the average sum of distances between each test data sequence in that target data class and other test data sequences can be calculated first. Then, the test data sequence with the smallest average distance sum can be selected as the core data sequence representing that target data class, i.e., as the target data sequence for that target data class.

[0198] In one embodiment, the target data sequence can be determined using the following formula:

[0199]

[0200] in, and These represent different sequences of data to be tested; euclidean() represents the Euclidean distance.

[0201] To better illustrate the data processing method provided in the above embodiments, a detailed description is given below using specific examples:

[0202] In a specific example, such as Figure 17 As shown, Figure 17This is the main flowchart of the data processing method provided in this example, based on, for example... Figure 17 The main flowchart shown illustrates the following steps in this data processing method:

[0203] First, the test data sequences collected from the network are preprocessed, mainly including missing value imputation and data standardization, to form time series datasets of equal length;

[0204] Then, the moving average method is used to extract the baseline for each data sequence to be tested, forming a baseline dataset;

[0205] Next, clustering was performed on the baseline dataset using the DBSCAN algorithm and the Euclidean distance metric, and a heuristic algorithm was used to automatically fine-tune the threshold parameters.

[0206] Then, for each category in the clustering results, a core sequence is determined using a distance metric;

[0207] Next, an anomalous data segment template is determined for a single core sequence;

[0208] Then, the data search space is automatically generated for each sequence;

[0209] Next, the system automatically obtains the corresponding abnormal data segment template based on the core sequence of the category to which each sequence belongs, and completes the search for similar abnormal segments in the data search space of the sequence to obtain N similar abnormal segments with a high degree of similarity, and anomaly labeling is performed on these N similar abnormal segments;

[0210] Finally, the annotation results of similar abnormal segments in each sequence are manually checked and locally corrected to form the final labeled abnormal data.

[0211] Furthermore, another embodiment of the present invention provides an apparatus comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor.

[0212] The processor and memory can be connected via a bus or other means.

[0213] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0214] It should be noted that the device in this embodiment may include, for example, Figure 1 The system architecture platform shown in the embodiment, the device in this embodiment, and such Figure 1 The system architecture platforms in the illustrated embodiments belong to the same inventive concept, and therefore have the same implementation principle and technical effect, which will not be described in detail here.

[0215] The non-transitory software program and instructions required to implement the data processing method of the above embodiments are stored in memory. When executed by a processor, the data processing method of the above embodiments is executed, for example, the method described above. Figure 2 Method steps S100 to S500 Figure 3 Method steps S310 to S330 in the text Figure 4 Method steps S311 to S312 in the text Figure 5 Method steps S410 to S430 in the text Figure 6 Method steps S432 to S434 in the text Figure 7 Method steps S110 to S130 in the text Figure 8 Method steps S600 to S700 in the text Figure 9 Method steps S610 to S630 in the text Figure 10 Method steps S611 to S612 in the text Figure 11 Method steps S710 to S730 in the text Figure 12 Method steps S732 to S734 in the text Figure 13 Method steps S121 to S123 in the text Figure 14 Method steps S1211 to S1212 in the text Figure 15 Method steps S810 to S840 in the text Figure 16 Method steps S901 to S913.

[0216] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0217] Furthermore, another embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions that are executed by a processor or controller, for example, by a processor in the above-described device embodiment, causing the processor to perform the data processing methods described above, such as executing the methods described above. Figure 2 Method steps S100 to S500 Figure 3Method steps S310 to S330 in the text Figure 4 Method steps S311 to S312 in the text Figure 5 Method steps S410 to S430 in the text Figure 6 Method steps S432 to S434 in the text Figure 7 Method steps S110 to S130 in the text Figure 8 Method steps S600 to S700 in the text Figure 9 Method steps S610 to S630 in the text Figure 10 Method steps S611 to S612 in the text Figure 11 Method steps S710 to S730 in the text Figure 12 Method steps S732 to S734 in the text Figure 13 Method steps S121 to S123 in the text Figure 14 Method steps S1211 to S1212 in the text Figure 15 Method steps S810 to S840 in the text Figure 16 Method steps S901 to S913.

[0218] It will be understood by those skilled in the art that all or some of the steps and systems in the methods disclosed above can be implemented as software, firmware, hardware, and suitable combinations thereof. Some or all of the physical components can be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit. Such software can be distributed on a computer-readable medium, which can include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, as is known to those skilled in the art, communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0219] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. A data processing method, comprising, obtaining a plurality of to-be-tested data sequences; performing clustering processing on the plurality of to-be-tested data sequences to obtain target data classes; in each of the target data classes, calculating a distance average sum of each of the to-be-tested data sequences and the rest of the to-be-tested data sequences respectively, and determining the to-be-tested data sequence corresponding to the minimum value in the distance average sum as a target data sequence; obtaining a first abnormal data segment in the target data sequence; obtaining a first abnormal feature value of the target data sequence; determining a first data position corresponding to the first abnormal feature value in the target data sequence according to the first abnormal feature value; obtaining a first data search space according to the first data position; obtaining a second abnormal data segment corresponding to the first abnormal data segment in the first data search space according to the first abnormal data segment; annotating the second abnormal data segment.

2. The data processing method according to claim 1, characterized in that, The obtaining of the first abnormal feature value of the target data sequence comprises: obtaining a first baseline prediction data of the target data sequence; obtaining a first abnormal feature value according to the deviation value of the first baseline prediction data and the data in the target data sequence.

3. The data processing method of claim 1, wherein, The obtaining of the second abnormal data segment corresponding to the first abnormal data segment in the first data search space according to the first abnormal data segment comprises: determining a third data segment in the first data search space; performing similarity calculation on the first abnormal data segment and the third data segment to obtain a first similarity value corresponding to the third data segment; determining the corresponding third data segment as a second abnormal data segment according to the first similarity value.

4. The data processing method according to claim 3, characterized in that, The determining of the corresponding third data segment as a second abnormal data segment according to the first similarity value comprises: when the first similarity value is less than a preset threshold, determining the third data segment corresponding to the first similarity value as a second abnormal data segment.

5. The data processing method according to claim 3, characterized in that, When the number of the third data segments is more than two, the determining of the corresponding third data segment as a second abnormal data segment according to the first similarity value comprises: obtaining the first similarity value less than the preset threshold; sorting the first similarity value less than the preset threshold from small to large to adjust the sorting of the corresponding third data segment; determining the first N third data segments as second abnormal data segments, wherein N is greater than or equal to 1.

6. The data processing method of claim 1, wherein, Further comprising: obtaining a second data search space in each of the rest of the to-be-tested data sequences in each of the target data classes respectively; obtaining the second abnormal data segment in the second data search space in the rest of the to-be-tested data sequences respectively by using the first abnormal data segment in the target data sequence.

7. The data processing method according to claim 6, characterized in that, The obtaining of a second data search space in each of the rest of the to-be-tested data sequences in each of the target data classes comprises: obtaining a second abnormal feature value of the rest of the to-be-tested data sequences in each of the target data classes respectively; determining a second data position corresponding to the second abnormal feature value in the rest of the to-be-tested data sequences respectively according to the second abnormal feature value; According to the second data position, a second data search space of the remaining to-be-tested data sequence is obtained respectively.

8. The data processing method according to claim 7, characterized in that, The second abnormal feature value of the remaining to-be-tested data sequence in each target data class comprises: According to the second baseline prediction data, a second abnormal feature value of the remaining to-be-tested data sequence is obtained respectively according to the deviation value of the data in the remaining to-be-tested data sequence. The second abnormal data segment in the remaining to-be-tested data sequence is obtained respectively in the second data search space of the remaining to-be-tested data sequence by using the first abnormal data segment in the target data sequence, comprising:

9. The data processing method according to claim 6, characterized in that, A fourth data segment in the second data search space of the remaining to-be-tested data sequence is determined respectively; The first abnormal data segment in the target data sequence is subjected to similarity calculation with the fourth data segment in the remaining to-be-tested data sequence respectively, so as to obtain a second similarity value corresponding to the fourth data segment; According to the second similarity value, the corresponding fourth data segment in the remaining to-be-tested data sequence is determined as a second abnormal data segment in the remaining to-be-tested data sequence. When the second similarity value is less than a preset threshold value, the corresponding fourth data segment in the remaining to-be-tested data sequence is determined as a second abnormal data segment in the remaining to-be-tested data sequence.

10. The data processing method according to claim 9, characterized in that, When the number of the fourth data segments is more than two, according to the second similarity value, the corresponding fourth data segment in the remaining to-be-tested data sequence is determined as a second abnormal data segment in the remaining to-be-tested data sequence, comprising: The second similarity value corresponding to the remaining to-be-tested data sequence is obtained respectively, and the value is less than a preset threshold value; 11. The data processing method according to claim 9, characterized in that, The second similarity value with a value less than a preset threshold value is sorted from small to large, so as to adjust the sorting of the corresponding fourth data segment in the remaining to-be-tested data sequence respectively; The first N fourth data segments in the remaining to-be-tested data sequence are determined as second abnormal data segments respectively, wherein N is greater than or equal to 1. The target data class is obtained by clustering processing of the plurality of to-be-tested data sequences, comprising: The plurality of to-be-tested data sequences are subjected to data preprocessing respectively, so as to obtain a plurality of first preprocessing data sequences; 12. The data processing method of claim 1, wherein, The plurality of first preprocessing data sequences are subjected to baseline extraction processing respectively, so as to obtain a plurality of second preprocessing data sequences; The plurality of second preprocessing data sequences are clustered according to similarity, so as to obtain a target data class. The plurality of to-be-tested data sequences are subjected to data preprocessing respectively, so as to obtain a plurality of first preprocessing data sequences, comprising: The plurality of to-be-tested data sequences are subjected to missing value filling processing respectively, so as to obtain a plurality of filling data sequences; 13. The data processing method according to claim 12, characterized in that, The plurality of filling data sequences are subjected to data standardization processing respectively, so as to obtain a plurality of first preprocessing data sequences. ​ ​ 14. The data processing method according to claim 12, characterized in that, The clustering of the plurality of second pre-processed data sequences by similarity comprises: The clustering of the plurality of second pre-processed data sequences by similarity comprises:

15. The data processing method according to claim 14, characterized in that, The neighborhood distance threshold is obtained by a heuristic algorithm, wherein the heuristic algorithm comprises the following steps: The similarity matrix data is obtained by calculating the similarity between each two of the plurality of second pre-processed data sequences by the distance function; The k-dist sequence is obtained by calculating the k-dist distance based on the similarity matrix data; The initial distance threshold parameter is obtained based on the k-dist sequence; The initial distance threshold parameter is adjusted to obtain the neighborhood distance threshold.

16. The data processing method according to claim 15, characterized in that, The initial distance threshold parameter is obtained based on the k-dist sequence, comprising: The slope of each k-dist point in the k-dist sequence and the two adjacent points before and after it is calculated, and when the slopes of the two adjacent points before and after it are both less than a preset slope threshold, and when the difference between the slopes of the two adjacent points before and after it is less than a preset slope difference threshold, the current k-dist point is determined as a candidate distance threshold; The largest one of the candidate distance thresholds is determined as the initial distance threshold parameter.

17. The data processing method according to claim 15 or 16, characterized in that, The initial distance threshold parameter is adjusted to obtain the neighborhood distance threshold, comprising: The step length is obtained; The initial distance threshold parameter is adjusted to obtain the distance adjustment threshold according to the step length, and when the number of categories decreases, the distance adjustment threshold obtained in the previous step is determined as the neighborhood distance threshold.

18. An apparatus comprising: Memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method of any one of claims 1 to 17 when executing the computer program.

19. A computer readable storage medium storing computer executable instructions for performing the data processing method of any one of claims 1 to 17.

Images