Label determination method, device and system

Abstract

The present application discloses a label determination method, device and system, which belongs to the field of AI. The method includes: obtaining a target feature vector of a first time series, where a time series is a set of data arranged in time sequence; obtaining the similarity between the target feature vector and a reference feature vector in a reference feature vector set, where the reference feature vector is a feature vector of a second time series with a determined label; when the similarity between the target feature vector and the first reference feature vector is greater than a similarity threshold, determining the label corresponding to the first reference feature vector as the label of the first time series, where the first reference feature vector is a reference feature vector in the reference feature vector set. The present application improves the accuracy of label determination, and the present application is used to determine labels for machine learning models.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence (AI), and in particular to a label determination method, device, and system. Background Art

[0002] Machine learning involves training a machine learning model based on training samples, enabling the model to predict data beyond the training samples (e.g., predicting categories). As a key branch of AI, machine learning has been widely applied in numerous fields. Based on learning methods, machine learning algorithms can be categorized into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Supervised learning is a fundamental type of machine learning algorithm.

[0003] When training a machine learning model using a supervised learning algorithm, a large amount of sample data must first be manually labeled (for example, if the sample is a face image, the labeling process can include labeling a face image as "wearing glasses"). This labeled sample data is then used to train the machine learning model to adjust the parameters used by the machine learning model. The trained machine learning model can then be used to perform corresponding functions, such as image recognition or language translation. The labeling process of sample data is called the label determination process, and the labeled content is the label. The label is used to identify the data, such as the category of the data.

[0004] However, when using supervised learning algorithms or semi-supervised learning algorithms for model training, a large amount of manpower is required to label sample data, and the cost of determining labels is high. Summary of the Invention

[0005] The present invention provides a method, device, and system for determining a tag. This method can address the current problem of high tag determination costs. The technical solution is as follows:

[0006] In a first aspect, a tag determination method is provided, the method comprising:

[0007] Obtain the target feature vector of the first time series, where the time series is a set of data arranged in time sequence;

[0008] Obtaining similarity between the target feature vector and a reference feature vector in a reference feature vector set, where the reference feature vector is a feature vector of a second time series with a determined label;

[0009] When the similarity between the target feature vector and the first reference feature vector is greater than a similarity threshold, the label corresponding to the first reference feature vector is determined as the label of the first time series, and the first reference feature vector is a reference feature vector in the reference feature vector set.

[0010] The label determination method provided in the embodiments of the present application migrates labels based on the similarity of the feature vectors of the time series, enabling automatic labeling of sample data and reducing the cost of label determination. Furthermore, because the similarity calculation is related to the feature vectors of the time series, the influence of interference information inherent in the time series itself is avoided. For example, the influence of interference information such as sampling period, amplitude variation, quadrant drift, and noise can be reduced. This improves the accuracy of label determination. In particular, label migration can still be performed accurately in high-dimensional time series.

[0011] Moreover, applying the label determination method provided in the embodiments of the present application to scenarios such as supervised learning algorithms or semi-supervised learning algorithms that require a large amount of labeled sample data can effectively reduce the labeling cost and improve the modeling efficiency of the machine learning model.

[0012] Optionally, the first time series is a time series of a network KPI.

[0013] Optionally, the reference feature vector includes data of one or more features, the target feature vector includes data of one or more features, the similarity between the target feature vector and the first reference feature vector is the similarity between the first feature vector and the second sub-feature vector, and the first sub-feature vector and the second sub-feature vector are respectively composed of data corresponding to the same features in the target feature vector and the first reference feature vector.

[0014] In the embodiments of the present application, the reference feature vectors and target feature vectors included in the reference feature vector set may be obtained using the same extraction algorithm or different extraction algorithms. Accordingly, the categories and number of features involved in each reference feature vector and target feature vector may be different. Therefore, it is necessary to perform corresponding processing for different situations.

[0015] In the first case, the reference feature vector and the target feature vector have different feature categories and different numbers. The similarity determination process then includes: selecting the same first feature from the features corresponding to the target feature vector and the features corresponding to the first reference feature vector; obtaining the data corresponding to the first feature in the target feature vector to obtain a first sub-feature vector composed of the obtained data; obtaining the data corresponding to the first feature in the first reference feature vector to obtain a second sub-feature vector composed of the obtained data; and determining the similarity between the first sub-feature vector and the second sub-feature vector. The similarity between the first sub-feature vector and the second sub-feature vector is the similarity between the reference feature vector and the target feature vector.

[0016] In the first case, by screening the first sub-eigenvector and the second sub-eigenvector and calculating the similarity between the two as the similarity between the reference eigenvector and the target eigenvector, the similarity calculation process can be simplified and the accuracy of the similarity finally calculated can be ensured.

[0017] In the second case, the reference feature vector and the target feature vector involve the same feature categories and number. The reference feature vector and the target feature vector can be directly obtained as a first sub-feature vector and a second sub-feature vector, respectively. The similarity between the first sub-feature vector and the second sub-feature vector is determined, and the similarity between the first sub-feature vector and the second sub-feature vector is the similarity between the reference feature vector and the target feature vector.

[0018] In the second case, by setting the categories and number of features involved in the reference feature vector and the target feature vector to be the same, the feature screening process can be reduced and the similarity calculation process can be further simplified.

[0019] In the above two cases, the first sub-feature vector and the second sub-feature vector are both represented in sequence form, the data at the same position in the first sub-feature vector and the second sub-feature vector correspond to features of the same category, and the similarity between the first sub-feature vector and the second sub-feature vector is negatively correlated with the distance between the first sub-feature vector and the second sub-feature vector.

[0020] Accordingly, the distance between the first sub-feature vector and the second sub-feature vector can be first obtained; then, based on the obtained distance, the similarity between the first sub-feature vector and the second sub-feature vector can be determined. For example, the distance can be calculated using the Euclidean distance formula, the Chebyshev distance formula, the cosine distance formula, the Mahalanobis distance formula, or other distance formulas.

[0021] Since the distance between the first sub-eigenvector and the second sub-eigenvector can effectively reflect the similarity between the two, the similarity can be quickly determined by calculating the distance, thereby improving the efficiency of similarity determination.

[0022] In an embodiment of the present application, when there is a feature vector whose similarity with each reference feature vector in the reference feature vector set is not greater than the similarity threshold, it is also necessary to determine the label by manual labeling to ensure that the feature vector for which the label needs to be determined can eventually be labeled with the corresponding label. In an embodiment of the present application, based on the different forms of the time series sent by the analysis device to the management device, the manual labeling process can be divided into an individual labeling process (in this scenario, the analysis device usually sends one time series to be labeled to the management device at a time) and a cluster labeling process (in this scenario, the analysis device usually sends a set of time series to be labeled to the management device at a time). The embodiment of the present application describes the manual labeling process in the following two optional ways:

[0023] In the first alternative, the manual annotation process includes the following individual annotation processes:

[0024] When the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, the first time series is sent to the management device so that the management device presents the first time series; and the label of the first time series sent by the management device is received.

[0025] By having professionals mark the labels of the first time series, it is possible to determine the labels of the first time series when the labels cannot be transferred.

[0026] In the second alternative, the manual labeling process includes the following cluster labeling process:

[0027] Obtain a first feature vector set, wherein the similarity between any feature vector in the first feature vector set and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, and the label of the time series corresponding to any feature vector is not determined; send the time series corresponding to the first feature vector set to a management device so that the management device can present the time series corresponding to the first feature vector set; receive the label of the time series corresponding to the first feature vector set sent by the management device.

[0028] By having professionals label the time series corresponding to the first set of feature vectors, labels can be determined even when the labels of the time series corresponding to the first set of feature vectors cannot be migrated. Furthermore, a single interaction with the management device can label multiple time series, saving network overhead.

[0029] There are many ways to implement the analysis device sending the time series corresponding to the first feature vector set to the management device. This embodiment of the application uses the following two implementations as examples for illustration:

[0030] In the first implementation method, the analysis device sends the time series corresponding to the first feature vector set to the management device. After receiving the time series, the management device presents the time series corresponding to the first feature vector set, and professionals mark the labels of the time series corresponding to the first feature vector set.

[0031] In a second implementation method, before sending the time series corresponding to the first feature vector set to the management device, the analysis device may also first perform clustering processing on the feature vectors in the first feature vector set to obtain the category relationship of the feature vectors in the first feature vector set; and then when sending the time series corresponding to the first feature vector set to the management device, the category relationship is also sent to the management device at the same time, so that the management device can present the time series corresponding to the first feature vector set according to the category relationship.

[0032] For example, the management device can display multiple time series belonging to the same category on the same user page, and multiple time series belonging to different categories on different user pages; for another example, the management device can display multiple time series belonging to different categories at different locations on the same user page; for another example, the management device can display each time series in correspondence with the category to which it belongs. The management device presents the time series corresponding to the first feature vector set according to the category relationship, which can be used by professionals to refer to the category relationship when labeling, thereby assisting professionals in labeling. Based on this, professionals can label time series belonging to the same category with the same label, thereby improving labeling efficiency and increasing labeling accuracy.

[0033] Optionally, clustering the feature vectors in the first feature vector set includes:

[0034] Based on the distance between every two feature vectors in the first feature vector set, counting the neighbor vectors of each feature vector, where the neighbor vectors of any feature vector in the first feature vector set are other feature vectors in the first feature vector set whose distance from the any feature vector is less than a distance threshold, where the distance threshold is a distance specified among multiple distances determined based on the first feature vector set;

[0035] Based on the statistical results, every two feature vectors whose number of identical neighbor vectors is greater than a threshold value are classified as feature vectors of the same type. For example, the threshold value is a number specified in the number of neighbor vectors of each feature vector in the first feature vector set.

[0036] Since the distance threshold and the quantity threshold are relatively changing values, the category relationship finally obtained based on these two thresholds is more accurate, can better reflect the correlation between each feature vector, and improve the adaptability of the clustering algorithm.

[0037] In an embodiment of the present application, when the label of a time series corresponding to a feature vector is determined, the feature vector can be added to the reference feature vector set as a reference basis for label migration. However, the labels corresponding to some feature vectors may be erroneous due to human error or machine algorithm error. If these feature vectors are added to the reference feature vector set, it is easy to cause label conflicts in the label migration process. For example, there are multiple reference feature vectors whose similarity with the target feature vector of a certain time series is greater than the similarity threshold, and the labels are different, resulting in the inability to perform label migration on the certain time series. Therefore, it is necessary to perform conflict detection processing on the feature vectors added to the reference feature vector set to avoid adding feature vectors with incorrect labels to the reference feature vector set. For example, the conflict detection process may include the following steps:

[0038] Obtaining a first feature vector of a third time series with a determined label;

[0039] Obtaining similarity between the first feature vector and a reference feature vector in the reference feature vector set;

[0040] When the similarity between the first feature vector and each reference feature vector in the reference feature vector set is not greater than a storage similarity threshold, the first feature vector is added to the reference feature set as a reference feature vector.

[0041] Optionally, the method further includes:

[0042] When the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is the same as the label corresponding to the second feature vector, the first feature vector is added to the reference feature set as a reference feature vector.

[0043] Optionally, the method further includes:

[0044] When the similarity between the first feature vector and a second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is different from the label corresponding to the second feature vector, sending the time series corresponding to the first feature vector and the time series corresponding to the second feature vector to the management device, so that the management device presents the time series corresponding to the first feature vector and the time series corresponding to the second feature vector;

[0045] receiving, from the management device, the same label of the time series corresponding to the first feature vector and the time series corresponding to the second feature vector;

[0046] Based on the received label, updating the pre-stored label of the time series corresponding to the first feature vector and the label of the time series corresponding to the second feature vector;

[0047] The first feature vector is added as a reference feature vector to the reference feature set.

[0048] Optionally, both the target feature vector and the reference feature vector include data of one or more features among statistical features, fitting features or frequency domain features.

[0049] Optionally, the label determination method provided in the embodiment of the present application is applied in an anomaly detection scenario and can automatically determine labels. In this application scenario, the aforementioned label determination method is executed by a network analyzer, and the label corresponding to the reference feature vector is an anomaly detection label. In the anomaly detection scenario, time series data includes network key performance indicators (KPIs), and network KPIs include network traffic KP, network service KPIs, etc. Among them, network equipment KPIs can be central processing unit (CPU) utilization, optical power, etc., and network service KPIs can be network traffic, packet loss rate, latency, number of user accesses, etc. Among them, network traffic KPI is time series data with periodicity. Since a large number of KPI anomalies have similar characteristics, the label determination method provided in the embodiment of the present application is applied to an anomaly detection scenario, and can automatically migrate labels within a certain range, improve label utilization, reduce labeling costs, and the accuracy of the determined labels is higher than that of traditional label migration methods.

[0050] In a second aspect, a tag determination device is provided, comprising: multiple functional modules, wherein the multiple functional modules interact with each other to implement the method of the first aspect and its respective embodiments. The multiple functional modules can be implemented based on software, hardware, or a combination of software and hardware, and the multiple functional modules can be arbitrarily combined or divided based on the specific implementation.

[0051] In a third aspect, a label determination device is provided, comprising: a processor and a memory;

[0052] The memory is used to store a computer program, wherein the computer program includes program instructions;

[0053] The processor is configured to call the computer program to implement any label determination method as described in the first aspect.

[0054] In a fourth aspect, a computer storage medium is provided, on which instructions are stored. When the instructions are executed by a processor, the label determination method as described in any one of the first aspects is implemented.

[0055] In a fifth aspect, a chip is provided, which includes a programmable logic circuit and / or program instructions. When the chip is running, it implements the label determination method as described in any one of the first aspects.

[0056] In a sixth aspect, a computer program product is provided, wherein instructions are stored in the computer program product. When the instructions are executed on a computer, the computer is caused to execute the label determination method as described in any one of the first aspects.

[0057] The beneficial effects of the technical solutions provided in the embodiments of the present application include at least:

[0058] The label determination method provided in the embodiment of the present application migrates labels based on the similarity of the characteristic vectors of the time series, which can realize automatic labeling of sample data and reduce the cost of label determination. And because the similarity calculation is related to the characteristic vector of the time series, the influence of interference information of the time series itself is avoided, for example, the influence of interference information such as sampling period, amplitude change, quadrant drift and noise can be reduced. The accuracy of label determination is improved. In particular, label migration can still be performed accurately in high-dimensional time series. Applying the label determination method provided in the embodiment of the present application to scenarios such as supervised learning algorithms or semi-supervised learning algorithms that require a large amount of labeled sample data can effectively reduce the labeling cost and improve the modeling efficiency of the machine learning model.

[0059] Furthermore, the label determination method provided in the embodiment of the present application uses the similarity of feature vectors for label migration, which is not limited to the label migration of time series with similar waveforms. Label migration can be performed as long as similarity in certain feature dimensions is ensured. Therefore, it can be seen that the embodiment of the present application can be applied to the label migration of time series with different waveforms. Therefore, it can expand the scenarios of label generalization, improve the flexibility and utilization of label migration, and reduce the modeling cost of machine learning models. In particular, in the anomaly detection scenario, label migration between KPIs with certain similar features can be achieved.

[0060] Furthermore, the analysis device clusters the first set of feature vectors to determine the category relationships. The management device then presents the time series corresponding to the first set of feature vectors based on these category relationships. This category relationship can be used by professionals as a reference during labeling, assisting them in labeling. This allows professionals to label time series belonging to the same category with the same label, improving labeling efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 This is a schematic diagram of an application scenario involved in a tag determination method provided in an embodiment of the present application;

[0062] Figure 2 This is a schematic diagram of another application scenario involved in a tag determination method provided in an embodiment of the present application;

[0063] Figure 3 This is a flowchart of a tag determination method provided in an embodiment of the present application;

[0064] Figure 4 1 is a flow chart of obtaining similarity between a target feature vector and a reference feature vector in a reference feature vector set, provided by an embodiment of the present application;

[0065] Figure 5 This is a flow chart of a conflict detection method provided by an embodiment of the present application;

[0066] Figure 6 This is a flow chart of another tag determination method provided in an embodiment of the present application;

[0067] Figure 7 This is a schematic diagram of a process for clustering feature vectors in a first feature vector set provided by an embodiment of the present application;

[0068] Figure 8 is a block diagram of a tag determination device provided by an illustrative embodiment of the present application;

[0069] Figure 9 is a block diagram of another tag determination device provided by an illustrative embodiment of the present application;

[0070] Figure 10 is a block diagram of another tag determination device provided by an illustrative embodiment of the present application;

[0071] Figure 11 is a block diagram of another tag determination device provided by an illustrative embodiment of the present application;

[0072] Figure 12 is a block diagram of a tag determination device provided in another exemplary embodiment of the present application;

[0073] Figure 13 is a block diagram of another label determination device provided by another exemplary embodiment of the present application;

[0074] Figure 14 is a block diagram of another label determination device provided by another exemplary embodiment of the present application;

[0075] Figure 15This is a block diagram of a label determination device provided in another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0076] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0077] To facilitate readers' understanding, the embodiments of the present application briefly introduce the machine learning algorithm involved in the provided label determination method.

[0078] Machine learning algorithms, as a key branch of AI, have been widely applied in numerous fields. Based on their learning methods, machine learning algorithms can be categorized into supervised learning, unsupervised learning, semi-supervised learning, and reinforcement learning. Supervised learning algorithms are algorithms that learn an algorithm or establish a pattern based on training data and then use this algorithm or pattern to infer new instances. Training data, also known as sample data, consists of input data and expected outputs. The expected output of a machine learning algorithm model is called a label, which can be a continuous value (called a regression or regression label) or a predicted classification result (called a classification label). Unsupervised learning algorithms differ from supervised learning algorithms in that their sample data is not labeled. Machine learning algorithms analyze the characteristics of the data to achieve specific results. Semi-supervised learning algorithms use sample data that is partially labeled and partially unlabeled, with unlabeled data far outnumbering labeled data. Reinforcement learning algorithms maximize expected benefits by repeatedly experimenting within an environment. Rewards and penalties imposed by the environment help them choose the option that maximizes benefits. Supervised learning algorithms are a fundamental type of machine learning algorithm, capable of achieving excellent results in applications such as image recognition and language translation, given sufficient data. However, the cost of obtaining labels for supervised learning algorithms is high, requiring significant manpower to annotate samples. Many application scenarios lack sufficient labeled data (i.e., labeled sample data).

[0079] As mentioned above, when using supervised learning algorithms or semi-supervised learning algorithms for model training, a large amount of manpower is required to label sample data, and the cost of determining labels is high.

[0080] A label determination method is currently proposed. This method uses label migration (also known as label generalization) to determine labels. That is, the label of a time series with a determined label is migrated to another time series similar to the time series as the label of the other time series. Among them, a time series is a set of data arranged in a time sequence. The time sequence is usually the order in which the data is generated. A time series is a data form of sample data. The data in a time series is also called a data point. For example, the time series X is X=(x1,x2,…,x n ), then the time series has n data points, from x1 to x n , the length of the time series is n.

[0081] Assuming that the first time series is the time series for which a label is to be determined, the label determination process includes: obtaining waveform similarity between the first time series and multiple reference time series; when the waveform similarity between the first time series and one of the multiple reference time series is greater than a waveform similarity threshold, determining the label corresponding to the reference time series as the label of the first time series. However, this method of label migration by comparing the waveform similarity of time series is easily affected by various interference information inherent in the time series (such as sampling period, amplitude change, quadrant drift, and noise), and the accuracy of label determination is relatively low.

[0082] Furthermore, a label determination method based on dynamic time warping (DTW) has been proposed. When the lengths of the first time series and the reference time series are different (i.e., the number of data points in the sequence), the correspondence between the two time series is established by regularizing the time axis, and then the waveform similarity between the two is calculated, thereby reducing the influence of sampling period, amplitude change, and quadrant drift to a certain extent. However, the algorithm for regularizing the time axis in this label determination method is complex, and it still cannot avoid the influence of noise in the time series. In particular, its practicality is low in high-dimensional time series.

[0083] This embodiment of the present application provides a label determination method that transfers labels based on the similarity of the feature vectors of a time series. The similarity calculation is related to the feature vectors of the time series, avoiding the influence of interference information inherent in the time series and improving the accuracy of label determination. This method can accurately transfer labels, especially in high-dimensional time series.

[0084] Please refer to Figure 1 , Figure 1 This is a schematic diagram of an application scenario involved in the tag determination method provided in the embodiment of this application. Figure 1As shown, the application scenario includes an analysis device 101, a management device 102, and network devices 103a to 103c (collectively referred to as network devices 103). Figure 1 The number of analysis devices, management devices, and network devices is for illustration only and does not limit the application scenarios involved in the label determination method provided in the embodiments of the present application. The network involved in the application scenario may be a second-generation (2G) communication network, a third-generation (3G) communication network, a long-term evolution (LTE) communication network, or a fifth-generation (5G) communication network.

[0085] Among them, the analysis device 101, the management device 102 and the network device 103 can be deployed on the same device, or they can be deployed on different devices respectively. For example, when the analysis device 101, the management device 102 and the network device 103 are deployed on different devices, the analysis device 101 can be a server, or a server cluster composed of several servers, or a cloud computing service center. The management device 102 can be a computer, or a server, or a server cluster composed of several servers, or a cloud computing service center, and the management device 102 can be an operations support system (OSS) or other network device connected to the analysis device. The network device 103 can be a router, a switch, a base station, etc., which can be a network device of the core network or a network device of the edge network. The analysis device 101 is connected to the network device 103 and the management device 102 respectively through a wired network or a wireless network.

[0086] The network device 103 is used to upload the collected data, such as various types of time series data, to the analysis device 101. The analysis device 101 is used to extract and use data from the network device 103, such as determining the labels of the acquired time series. The management device 103 is used to manage the analysis device 101. Optionally, the data uploaded by the network device 103 to the analysis device 101 may also include various types of log data and device status data. The analysis device 101 is also used to train one or more machine learning models. Different machine learning models can respectively implement functions such as anomaly detection, prediction, network security protection, and application identification using the data uploaded by the network device 103. The analysis device can also implement feature selection and automatic updating of each machine learning model, and feed back the selected features and the update results of the model to the management device 102, which decides whether to retrain the model. Corresponding to different machine learning models, the analysis device 101 can determine different labels using the label determination method provided in this application.

[0087] Optionally, the above application scenario may not include the network device 103, and the analysis device 101 may also receive time series data input by the management device 103. The embodiment of the present application only schematically illustrates the source of the time series data and does not limit it.

[0088] Furthermore, the label determination method provided in the embodiment of the present application can be used in anomaly detection scenarios. Anomaly detection refers to the detection of patterns, data, or time that do not conform to predictions. Traditional anomaly detection is performed by professionals (also known as experts) who study historical data and then find anomalies, that is, label the abnormal data with an "abnormal" label. The data sources for anomaly detection include applications, processes, operating systems, devices, or networks. With the increase in the complexity of computing systems, manual work is no longer capable of meeting the current difficulty of anomaly detection.

[0089] The label determination method provided in the embodiment of the present application is applied in the anomaly detection scenario and can automatically determine the label. Figure 2 , Figure 2 This is a schematic diagram of an application scenario of anomaly detection involved in the label determination method provided in an embodiment of the present application. In this application scenario, the analysis device 101 can be a network analyzer, the management device 102 can be a controller, the machine learning model maintained by the analysis device 101 is an anomaly detection model, and the determined label is an anomaly detection label, which includes two classification labels: "normal" and "abnormal". Figure 1 Based on the scenario shown, the application scenario may also include a storage device 104, which is used to store the data provided by the network device 103. The storage device 104 may be a distributed storage device, and the analysis device 101 may read and write the data stored in the storage device 104. In this way, when the network device 103 has a lot of data, the storage device 104 stores the data, which can reduce the load of the analysis device 101 and improve the data analysis efficiency of the analysis device 101. It should be noted that when the amount of data provided by the network device 103 is small, the storage device 104 may not be set. In this case, the application scenario of anomaly detection can refer to Figure 1 The application scenario shown.

[0090] In anomaly detection scenarios, time series anomaly detection usually involves finding data points that are far away from a relatively established pattern or distribution. Time series anomalies include: sudden increases, sudden decreases, mean changes, etc. Time series anomaly detection algorithms include algorithms based on statistics and data distribution (such as the N-Sigma algorithm), algorithms based on distance / density (such as the local anomaly factor algorithm), isolation forest algorithms, or prediction-based algorithms (such as the Autoregressive Integrated Moving Average model (ARIMA) algorithm). The corresponding machine learning model can be a model based on statistics and data distribution (such as the N-Sigma model), a model based on distance / density (such as the local anomaly factor model), an isolation forest model, or a prediction-based model (such as ARIMA).

[0091] like Figure 1 and Figure 2 As mentioned above, the data uploaded by network device 103 includes various types of time series data, characterized by large data volumes and complex patterns and regularities. Therefore, when using this data for applications such as anomaly detection, prediction, classification, network security protection, application identification, or user experience evaluation (e.g., evaluating the user experience based on this data), a large number of machine learning models are used. Professionals need to annotate this data, which is a very large workload and extremely costly.

[0092] An embodiment of the present application provides a label determination method that can perform label migration, thereby reducing the labeling cost. Since the label migration is performed based on the similarity of the characteristic vectors of the time series, the similarity calculation is related to the characteristic vectors of the time series, avoiding the influence of interference information of the time series itself and improving the accuracy of label determination.

[0093] In the anomaly detection scenario, time series data includes network key performance indicators (KPIs), which include network equipment KPIs and network service KPIs. Network equipment KPIs can be central processing unit (CPU) utilization, optical power, etc., while network service KPIs can be network traffic, packet loss rate, latency, number of user accesses, etc. Network traffic KPIs are time series data with periodicity. For example, Figure 2In the anomaly detection scenario shown in , a machine learning model is used to detect anomalies in network traffic KPIs. Because many KPI anomalies have similar characteristics, the label determination method provided in the embodiments of this application can be applied to anomaly detection scenarios. This method can automatically migrate labels within a certain range, improving label utilization and reducing labeling costs. Furthermore, compared to traditional label migration methods, the determined labels are more accurate.

[0094] The embodiment of the present application provides a label determination method, which can be executed by the aforementioned analysis device. Assuming that the first time series is a sequence for which label determination is required, such as Figure 3 As shown, the method includes:

[0095] Step 301: Obtain a target feature vector of a first time series.

[0096] A time series is a collection of data arranged in a chronological order. This chronological order typically represents the order in which the data was generated. The data in a time series is also called data points. The time intervals between data points in a time series are typically constant, so time series can be analyzed and processed as discrete-time data. For example, the first time series can be a time series of network KPIs.

[0097] In one optional example, the analysis device can receive a time series sent by a network device or a management device; in another optional example, the analysis device has an input and output (I / O) interface, through which the time series is received; in yet another optional example, the analysis device can read the time series from a storage device.

[0098] The target feature vector is a vector that represents the characteristics of the first time series. It includes data of one or more features. That is, the target feature vector corresponds to a one-dimensional or multi-dimensional feature. The dimension of the feature corresponding to the target feature vector is the same as the number of data in the target feature vector (i.e., the feature has a one-to-one correspondence with the data). The feature refers to the characteristics of the first time series, which may include data features and / or extracted features.

[0099] Among them, data features are the inherent features of the data in the time series. For example, data features include data arrangement period, data change trend or data fluctuation, etc. Correspondingly, the data of data features include: data of data arrangement period, data change trend data or data fluctuation data, etc. The data arrangement period refers to the period involved in the data arrangement in the time series if the data in the time series is arranged periodically. For example, the data of the data arrangement period includes the period duration (that is, the time interval between the initiation of two periods) and / or the number of periods; the data change trend data is used to reflect the changing trend of the data arrangement in the time series (that is, the data change trend), for example, the data includes: continuous growth, continuous decline, first rise and then fall, first fall and then rise, or satisfying the normal distribution, etc.; the data fluctuation data is used to reflect the fluctuation state of the data in the time series (that is, data fluctuation), for example, the data includes a function that characterizes the fluctuation curve of the time series, or a specified value of the time series, such as the maximum value, minimum value or average value.

[0100] Extracting features refers to the process of extracting features from the data in the time series. For example, extracted features include statistical features, fitting features, or frequency domain features, and correspondingly, the data for extracting features includes statistical feature data, fitting feature data, or frequency domain feature data. Statistical features refer to the statistical characteristics of a time series. Statistical features are divided into quantitative features and attribute features. Quantitative features are further divided into measurement features and counting features. Quantitative features can be directly represented by numerical values. For example, the consumption values ​​of various resources such as CPU, memory, and IO resources are measurement features. The number of anomalies and the number of normally working devices are counting features. Attribute features cannot be directly represented by numerical values, such as whether a device has an anomaly or whether a device has downtime. The features in statistical features are indicators that need to be examined during statistics. For example, the statistical feature data includes a moving average (Moving_average), a weighted average (Weighted_mv), etc.; the fitting feature is the feature when the time series is fitted, and the fitting feature data is used to reflect the features of the time series used for fitting, for example, the fitting feature data includes the algorithm used when fitting, such as ARIMA; the frequency domain feature is the feature of the time series in the frequency domain, and the frequency domain feature is used to reflect the features of the time series in the frequency domain. For example, the frequency domain feature data includes: data on the law followed by the distribution of the time series in the frequency domain, such as the proportion of high-frequency components in the time series. Optionally, the frequency domain feature data can be obtained by performing wavelet decomposition on the time series.

[0101] The process of obtaining the target feature vector of the first time series may include: first determining a target feature to be extracted, and then extracting data of the determined target feature from the first time series to obtain the target feature vector. For example, the target feature to be extracted is determined based on an application scenario involved in the label determination method. In an optional example, the target feature is a preconfigured feature, such as a feature configured by a user.

[0102] In another optional example, the target feature is one or more of the specified features, for example, the specified feature is the aforementioned statistical feature.

[0103] It is worth noting that the user can pre-set specified features, but for the first time series, it may not have all the specified features. The analysis device can filter the features belonging to the specified features in the first time series as target features. For example, the target features include statistical features: time series decompose_seasonal (Tsd_seasonal), moving average, weighted average, time series classification, maximum, minimum, quantile, variance, standard deviation, year on year (yoy, referring to comparison with the same historical period), daily volatility, bucket entropy, sample entropy, sliding average, exponential sliding average, Gaussian distribution features or T distribution features, etc. One or more of the following, and accordingly, the target feature data includes data of the one or more statistical features;

[0104] And / or, the target feature includes a fitting feature: one or more of an autoregressive fitting error, a Gaussian process regression fitting error, or a neural network fitting error, and accordingly, the target feature data includes data of the one or more fitting features; and / or, the target feature includes a frequency domain feature: the proportion of high-frequency components in a time series; accordingly, the target feature data includes data of the proportion of high-frequency components in a time series, and the data can be obtained by performing wavelet decomposition on the time series.

[0105] Step 302: Obtain similarity between the target feature vector and the reference feature vectors in the reference feature vector set. Execute step 303 or 304.

[0106] A reference feature vector set is pre-established in the analysis device. The reference feature vector set includes one or more reference feature vectors, each of which is a feature vector of the second time series for which a label has been determined. The labels may be manually annotated, determined using the label determination method provided in the embodiments of the present application, or determined using other algorithms, which are not limited in the embodiments of the present application.

[0107] The label and the second time series corresponding to each reference feature vector in the reference feature vector set may be stored in the reference feature vector set or in other storage spaces, as long as the corresponding label and the second time series can be obtained by querying the reference feature vector.

[0108] The reference feature vector is a vector that characterizes the characteristics of the second time series, which includes data of one or more characteristics. That is, the reference feature vector corresponds to a one-dimensional or multi-dimensional feature. The features involved in the reference feature vector may include data features and / or extracted features. The interpretation of the dimensions of the data, the number of features, and the corresponding feature vectors can refer to the interpretation of the aforementioned target feature vector. The process of obtaining the reference feature vector of each second time series can refer to the aforementioned process of obtaining the target feature vector of the first time series. This embodiment of the present application will not be described in detail.

[0109] Table 1 is a schematic illustration of the data stored in the reference feature vector set. In Table 1, the time series and labels corresponding to each reference feature vector in the reference feature vector set can be stored in the reference feature vector set. The sample data identification (identification, ID) in Table 1 is the reference feature vector of KPI_1, which includes data of 4 features, and the data of the 4 features are: moving average (Moving_average), weighted average (Weighted_mv), time series decompose_seasonal (time series decompose_seasonal, Tsd_seasonal) and period yoy. The time series corresponding to the reference feature vector is (x1, x2,..., xn), and the corresponding label is "abnormal". Table 1 assumes that the reference feature vector set stores data in a fixed format, and the features of the reference feature vectors stored therein can also be pre-set features, then the data of the reference feature vector set can all be stored in the format of Table 1. When the embodiment of the present application is actually implemented, the reference feature vector set can also have other forms, and the embodiment of the present application does not limit this.

[0110] Table 1

[0111]

[0112] In the embodiments of the present application, the reference feature vectors and target feature vectors included in the reference feature vector set may be obtained using the same extraction algorithm or different extraction algorithms. Accordingly, the categories and number of features involved in each reference feature vector and target feature vector may be different. Therefore, it is necessary to perform corresponding processing for different situations.

[0113] Assume that the first reference feature vector is a reference feature vector in the reference feature vector set, the first feature is the same feature as the feature corresponding to the target feature vector and the feature corresponding to the first reference feature vector, that is, the first feature is the intersection of the feature corresponding to the target feature vector and the feature corresponding to the first reference feature vector, the first sub-feature vector is a vector composed of data corresponding to the first feature in the target feature vector, and the second sub-feature vector is a vector composed of data corresponding to the first feature in the first reference feature vector, then the similarity between the target feature vector and the first reference feature vector is the similarity between the first sub-feature vector and the second sub-feature vector. The embodiment of the present application is illustrated by taking the following two cases as examples.

[0114] In the first case, the reference feature vector and the target feature vector involve different categories and numbers of features. Figure 4 As shown, the process of obtaining the similarity between the target feature vector and the reference feature vectors in the reference feature vector set may include the following steps:

[0115] Step 3021: Filter the same first features from the features corresponding to the target feature vector and the features corresponding to the first reference feature vector.

[0116] The first feature includes one or more features. The process of obtaining the first feature in the aforementioned step 3021 can be achieved by obtaining the intersection of the feature corresponding to the target feature vector and the feature corresponding to the first reference feature vector. For example, assuming that the target feature vector Q1 includes data corresponding to four features, namely features y1 to y4, and the four data are q1 to q4, that is, Q1 = (q1, q2, q3, q4), and the corresponding feature set Y1 satisfies: Y1 = (y1, y2, y3, y4); the first reference feature vector Q2 includes data corresponding to three features, namely features y1, y4, and y5, and the three data are p1, p4, and p5, that is, Q2 = (p1, p4, p5), and the corresponding feature set Y2 satisfies: Y2 = (y1, y4, y5). Then the first feature Y satisfies: Y = Y1 ∩ Y2, then Y = (y1, y4).

[0117] It is worth noting that the first feature can also be obtained in other ways, such as comparing the feature corresponding to the target feature vector with the feature corresponding to the first reference feature vector in sequence, and this embodiment of the application does not limit this.

[0118] Step 3022: Acquire data corresponding to the first feature in the target feature vector, and obtain a first sub-feature vector composed of the acquired data.

[0119] Still taking the example of step 3021 , the first sub-feature vector is Q11 = (q1, q4) in the target feature vector Q1 = (q1, q2, q3, q4).

[0120] Step 3023: Acquire data corresponding to the first feature in the first reference feature vector, and obtain a second sub-feature vector composed of the acquired data.

[0121] Still using the example of step 3021, the second sub-feature vector is Q21 = (p1, p4) in the first reference feature vector Q2 = (p1, p4, p5). It is worth noting that the number and arrangement of data in the first and second sub-feature vectors are consistent to ensure the accuracy of subsequent similarity calculations.

[0122] Step 3024: Determine the similarity between the first sub-feature vector and the second sub-feature vector.

[0123] In the embodiment of the present application, the first sub-feature vector and the second sub-feature vector are both represented in sequence form. The data at the same position in the first sub-feature vector and the second sub-feature vector correspond to features of the same category. The similarity between the first sub-feature vector and the second sub-feature vector can be measured by the distance between the two sub-feature vectors. This similarity is negatively correlated with the distance between the first sub-feature vector and the second sub-feature vector. In other words, the greater the similarity between the two sub-feature vectors, the smaller the distance; the smaller the similarity, the greater the distance.

[0124] Then, the distance between the first sub-feature vector and the second sub-feature vector may be obtained first; and then, based on the obtained distance, the similarity between the first sub-feature vector and the second sub-feature vector may be determined.

[0125] Optionally, the distance between the first sub-eigenvector and the second sub-eigenvector is used to characterize the distance between the target eigenvector and the first reference eigenvector. The distance between the first sub-eigenvector and the second sub-eigenvector can be obtained in a variety of ways, for example, by using the Euclidean distance formula, the Chebyshev distance formula, the cosine distance formula, the Mahalanobis distance formula or other distance formulas.

[0126] For example, assume that the first sub-eigenvector is x=(f x1 ,f x2 ,…,f xn ), the second sub-eigenvector is y=(f y1 ,f y2 ,…,f yn ), the Mahalanobis distance formula is used to calculate the distance D between the first sub-eigenvector and the second sub-eigenvector M (x,y)D M (x,y), then the Mahalanobis distance formula is as follows:

[0127]

[0128] Among them, Σ -1 is the covariance matrix, Σ -1=E[(XE[X])(XE(X)) T ]. Covariance matrix Σ -1 is a predetermined matrix, which can be calculated from the first sub-feature vector and data of features with the same dimension as the first sub-feature vector in the reference feature vector set.

[0129] In an embodiment of the present application, the similarity between the first sub-feature vector and the second sub-feature vector is negatively correlated with the distance between the two. Therefore, the similarity S between the first sub-feature vector and the second sub-feature vector can be determined based on the obtained distance D and the similarity calculation formula. In one optional manner, the similarity calculation formula is: S = a / D. Where a is a preset value. For example, a = 1. In another optional manner, the similarity calculation formula is: S = 1-f(D), where f(D) represents the normalization of the distance D.

[0130] In the second case, the categories and number of features involved in the reference feature vector and the target feature vector are the same. Then, the first sub-feature vector is the same as the target feature vector, and the second sub-feature vector is the same as the first reference feature vector. There is no need to perform the screening action of the aforementioned step 3021. The process of obtaining the similarity between the target feature vector and the reference feature vector in the reference feature vector set can be: directly determining the similarity between the target feature vector and the first reference feature vector, that is, first obtaining the distance between the target feature vector and the first reference feature vector; then, based on the obtained distance, determining the similarity between the target feature vector and the first reference feature vector. The process of determining the similarity between the target feature vector and the first reference feature vector can refer to the aforementioned step 3024. This embodiment of the present application will not be repeated here.

[0131] It is worth noting that the aforementioned reference feature vector and target feature vector involve data of multiple features, and the more data with the same features, the more the similarity finally calculated can reflect the correlation between the reference feature vector and the target feature vector from multiple angles, and the label determined based on this is more accurate.

[0132] Step 303: When the similarity between the target feature vector and the first reference feature vector is greater than a similarity threshold, the label corresponding to the first reference feature vector is determined as the label of the first time series.

[0133] The similarity threshold can be pre-set by the user or determined by the analysis device based on the current application scenario. When the similarity between the target feature vector and the first reference feature vector is greater than the similarity threshold, it indicates that the first time series and the second time series corresponding to the first reference feature vector have high feature similarity, and the first time series meets the label transfer condition. The label corresponding to the first reference feature vector can be determined as the label of the first time series.

[0134] For example, if the label corresponding to the first reference feature vector is “abnormal”, then the label of the first time series is also “abnormal”.

[0135] Step 304: When the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, the first time series is sent to the management device so that the management device presents the first time series.

[0136] When the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, it means that the first time series and the second time series corresponding to each reference feature vector in the reference feature vector set have low similarity in features, and the first time series does not meet the label migration condition. In this case, the label of the first time series corresponding to the target feature vector can be manually annotated. Therefore, the analysis device can send the first time series to the management device, which can be the management device 102 in the aforementioned application environment. After receiving the first time series, the management device presents the first time series, and a professional person annotates the label of the first time series.

[0137] Step 305: Receive the first time series label sent by the management device.

[0138] Referring to step 304 , after the professional labels the first time series, the management device receives the labeled labels and sends the labels to the analysis device. The analysis device receives the labels and stores them in correspondence with the first time series.

[0139] It is worth noting that when the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, if the importance of the first time series is lower than the preset threshold, for example, the first time series is a randomly acquired time series, since automatic label determination is impossible, the analysis device may not label the first time series, that is, not execute steps 304 and 305, delete the first time series, obtain a new time series as the first time series, and execute the above steps 301 to 303 again to achieve label determination for the time series that meets the label migration conditions. In this way, there is no need for manual participation, and labels can be determined for all time series that meet the label migration conditions.

[0140] Step 306: Perform conflict detection processing on the feature vectors that need to be added to the reference feature vector set.

[0141] In an embodiment of the present application, when the label of a time series corresponding to a feature vector is determined, the feature vector can be added to the reference feature vector set as a reference basis for label migration. However, the labels corresponding to some feature vectors may be erroneous due to human errors or machine algorithm errors. If these feature vectors are added to the reference feature vector set, it is easy to cause label conflicts in the label migration process. For example, there are multiple reference feature vectors whose similarity with the target feature vector of a certain time series is greater than the similarity threshold, and the labels are different, resulting in the inability to perform label migration on the certain time series. Therefore, it is necessary to perform conflict detection processing on the feature vectors added to the reference feature vector set to avoid adding feature vectors with incorrect labels to the reference feature vector set. For example, Figure 5 As shown, the conflict detection process may include the following steps:

[0142] Step 3061: Obtain a first feature vector of the third time series with a determined label.

[0143] The labels of the third time series can be manually annotated, or determined by the label determination method provided in the embodiment of the present application, or determined by other algorithms. For example, the labels of the third time series can be the labels determined in the aforementioned step 303 or step 305. Accordingly, the third time series is determined by the aforementioned first time series.

[0144] The process of obtaining the first feature vector of the third time series can refer to the process of obtaining the target feature vector of the first time series in the aforementioned step 301, and will not be repeated in this embodiment of the present application. It is worth noting that when the third time series is the aforementioned first time series, the aforementioned target feature vector can be directly used as the first feature vector to reduce the process of extracting feature vectors again and reduce the computational cost.

[0145] Step 3062: Obtain similarity between the first feature vector and a reference feature vector in the reference feature vector set.

[0146] Step 3062 can refer to the aforementioned step 302, and will not be described in detail in this embodiment of the present application.

[0147] In a reference feature set, one or more reference feature vectors can correspond to the same label. However, the following error scenario can easily occur: multiple substantially related reference feature vectors may correspond to different labels due to human error or machine algorithm errors. In other words, reference feature vectors that should correspond to the same label may correspond to different labels. To reduce the occurrence of such error scenarios, steps 3063, 3064, or 3065 can be performed after step 3062.

[0148] Step 3063: When the similarity between the first feature vector and each reference feature vector in the reference feature vector set is not greater than the storage similarity threshold, the first feature vector is added to the reference feature set as a reference feature vector.

[0149] When the similarity between the first feature vector and each reference feature vector in the reference feature vector set is not greater than the storage similarity threshold, it means that the first feature vector is not similar to each reference feature vector in the reference feature vector set. Correspondingly, the similarity between the third time series and the second time series corresponding to each reference feature vector in the reference feature vector set is low. It is a brand new time series, and the first feature vector can be added to the reference feature set as a reference feature vector.

[0150] It is worth noting that the storage similarity threshold can be pre-set by the user or determined by the analysis device based on the current application scenario, and can be the same as or different from the similarity threshold in the aforementioned step 303.

[0151] Step 3064: When the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is the same as the label corresponding to the second feature vector, the first feature vector is added to the reference feature set as a reference feature vector.

[0152] When the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, it means that the first feature vector is similar to the second feature vector and the two are related; when the label corresponding to the first feature vector is the same as the label corresponding to the second feature vector, it means that the two related feature vectors correspond to the same label, then the first feature vector meets the conditions for joining the reference feature vector set, and the first feature vector is added to the reference feature set as a reference feature vector.

[0153] Step 3065: When the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is different from the label corresponding to the second feature vector, the time series corresponding to the first feature vector and the time series corresponding to the second feature vector are sent to the management device so that the management device can present the time series corresponding to the first feature vector and the time series corresponding to the second feature vector. Execute step 3066.

[0154] When the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, it means that the first feature vector is similar to the second feature vector and the two are related; when the label corresponding to the first feature vector is different from the label corresponding to the second feature vector, it means that the two related feature vectors correspond to different labels, and the label of the first feature vector or the second feature vector is incorrect. The label of the first feature vector or the second feature vector can be manually annotated again to ensure the accuracy of the label. Therefore, the analysis device can send the time series corresponding to the first feature vector and the time series corresponding to the second feature vector to the management device, and the management device can be the management device 102 in the aforementioned application environment. After receiving the time series corresponding to the first feature vector and the time series corresponding to the second feature vector, the management device presents the received time series, and the professional person annotates the label of the presented time series. Since the feature vectors corresponding to the two time series are related, the labels of the two manually annotated time series are the same label.

[0155] It should be noted that the analysis device can also send the label corresponding to the first eigenvector and the label corresponding to the second eigenvector to the management device. The management device can synchronously present the received labels when presenting the received time series for reference by professionals, which can improve the accuracy of the final labeling to a certain extent.

[0156] Step 3066: Receive the same label of the time series corresponding to the first feature vector and the time series corresponding to the second feature vector from the management device, and execute step 3067.

[0157] After the professionals label the presented time series, the management device receives the labeled labels and sends the labels to the analysis device, which receives the labels.

[0158] Step 3067: Based on the received label, update the pre-stored label of the time series corresponding to the first eigenvector and the label of the time series corresponding to the second eigenvector. Execute step 3068.

[0159] Referring to step 3065, since the pre-stored labels of the time series corresponding to the first eigenvector and the pre-stored labels of the time series corresponding to the second eigenvector are different, the analysis device can update the pre-stored labels of the time series corresponding to the first eigenvector and the pre-stored labels of the time series corresponding to the second eigenvector based on the received labels, ensuring that the updated labels of the time series corresponding to the first eigenvector and the pre-stored labels of the time series corresponding to the second eigenvector are the same, thereby avoiding label conflicts.

[0160] Step 3068: Add the first feature vector as a reference feature vector to the reference feature set.

[0161] It is worth noting that the aforementioned steps 3061 to 3068 are only an illustrative implementation method for conflict detection. In actual implementation, the embodiment of the present application can also adopt other methods for conflict detection. For example, when the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is different from the label corresponding to the second feature vector, a professional can also manually perform conflict detection, and steps 3065 to 3068 can also be replaced by: presenting the first feature vector, the corresponding time series and label through the analysis device itself or the management device; and presenting the second feature vector, and the label of the corresponding time series; receiving a deletion instruction, the deletion instruction instructs to delete the first feature vector, the corresponding time series and label, or the deletion instruction instructs to delete the second feature vector, and the label of the corresponding time series; deleting the feature vector indicated by the deletion instruction, and the corresponding time series and label. If the analysis device receives the deletion instruction, it means that the feature vector indicated by the deletion instruction cannot play an effective reference role in the label migration process. By deleting the feature vector, label conflicts in the label migration process can be avoided.

[0162] The above-mentioned step 306 is explained by taking the example of performing conflict detection processing when the first feature vector is added to the reference feature vector set. When the embodiment of the present application is actually implemented, conflict detection processing can also be performed periodically, or conflict detection processing can be performed after receiving a detection trigger instruction. The conflict detection processing process includes: steps A1 to A6.

[0163] Step A1: Obtain any eigenvector of the reference eigenvector set as the third eigenvector.

[0164] Step A2: Obtain similarity between the third feature vector and other reference feature vectors in the reference feature vector set.

[0165] Step A2 can refer to the aforementioned step 302, and will not be described in detail in this embodiment of the present application.

[0166] Step A3: When the similarity between the third feature vector and any other reference feature vector in the reference feature vector set is not greater than the storage similarity threshold, any feature vector other than the third feature vector in the reference feature vector set is used as the third feature vector, and steps A1 to A7 are repeated until all feature vectors in the reference feature vector set are traversed, and the action is stopped.

[0167] Step A4: When the similarity between the third feature vector and the fourth feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the third feature vector is the same as the label corresponding to the fourth feature vector, any feature vector other than the third feature vector in the reference feature vector set is used as the third feature vector, and steps A1 to A7 are repeated until all feature vectors in the reference feature vector set are traversed and the action is stopped.

[0168] Step A5: When the similarity between the third feature vector and the fourth feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the third feature vector is different from the label corresponding to the fourth feature vector, the time series corresponding to the third feature vector and the time series corresponding to the fourth feature vector are sent to the management device so that the management device can present the time series corresponding to the third feature vector and the time series corresponding to the fourth feature vector. Execute Step A6.

[0169] Step A5 refers to the aforementioned step 3065, and this embodiment of the application will not be described in detail.

[0170] Step A6: Receive the same label of the time series corresponding to the third eigenvector and the time series corresponding to the fourth eigenvector sent by the management device, and proceed to step A7.

[0171] Step A6 refers to the aforementioned step 3066, and this embodiment of the application will not be described in detail.

[0172] Step A7: Based on the received label, update the pre-stored label of the time series corresponding to the third eigenvector and the label of the time series corresponding to the fourth eigenvector. Use any other eigenvector in the reference eigenvector set, except the third eigenvector, as the third eigenvector, and repeat steps A1 to A7 until all eigenvectors in the reference eigenvector set have been traversed, then stop.

[0173] By performing conflict detection within the reference feature vector set, label conflicts can be avoided, ensuring that the reference feature vectors in the reference feature vector set play an effective reference role. By deleting feature vectors that have no reference value, the accuracy of label determination can be improved.

[0174] In the aforementioned step 304, when the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, a first time series is sent to the management device, that is, each time the analysis device obtains a time series whose similarity with each reference feature vector in the reference feature vector set is not greater than the similarity threshold, the time series is sent to the management device for manual labeling. This labeling method is an individual labeling method, that is, a label is labeled during one interaction with the management device. When the embodiment of the present application is actually implemented, the manual labeling process can also have other implementation methods, such as a cluster labeling method, that is, labeling multiple labels during one interaction with the management device, such as Figure 6 As shown, when the cluster labeling method is adopted, the aforementioned steps 304 and 305 can be replaced by steps 307 to 309:

[0175] Step 307: Obtain a first feature vector set, wherein the similarity between any feature vector in the first feature vector set and each reference feature vector in the reference feature vector set is not greater than a similarity threshold, and the label of the time series corresponding to any feature vector is undetermined.

[0176] In one optional example, the number of feature vectors in the first feature vector set is a specified number. For example, after repeatedly performing steps 301 to 303, the analysis device obtains a specified number of fifth feature vectors and determines the specified number of fifth feature vectors as the first feature vector set. The similarity between the fifth feature vectors and each reference feature vector in the reference feature vector set is no greater than a similarity threshold, and the label of the time series corresponding to the fifth feature vector is undetermined. The fifth feature vector may include the aforementioned target feature vector.

[0177] In another optional example, the first feature vector set is a periodically acquired set. For example, during repeated execution of steps 301 to 303, the analysis device acquires a fifth feature vector at specified intervals to obtain the first feature vector set. The fifth feature vector is a feature vector whose similarity with each reference feature vector in the reference feature vector set within the most recent specified interval is no greater than a similarity threshold, and the label of the time series corresponding to the fifth feature vector is undetermined. The fifth feature vector may include the target feature vector.

[0178] In another optional example, the first feature vector set is a set obtained by the analysis device after receiving a collection instruction. For example, when the analysis device repeatedly performs the aforementioned steps 301 to 303 multiple times, if a collection instruction instructing to collect the fifth feature vector is received, then based on the collection instruction, the fifth feature vector is obtained to obtain the first feature vector set, and the fifth feature vector is a feature vector whose similarity with each reference feature vector in the reference feature vector set is not greater than the similarity threshold within the historical time length (the historical time length can be a specified time length, or the time length between the last collection instruction and the current collection instruction, or the time length specified in other ways), and the label of the time series corresponding to the fifth feature vector is not determined. The fifth feature vector may include the aforementioned target feature vector.

[0179] Step 308: Send the time series corresponding to the first feature vector set to the management device, so that the management device can present the time series corresponding to the first feature vector set.

[0180] In the first optional method, the analysis device sends the time series corresponding to the first feature vector set to the management device. After receiving the time series, the management device presents the time series corresponding to the first feature vector set, and professionals mark the labels of the time series corresponding to the first feature vector set.

[0181] For example, the management device can simultaneously display the time series corresponding to multiple feature vectors in the first feature vector set on the same user interface, or it can display the time series corresponding to multiple feature vectors in the first feature vector set separately in a scrolling manner. This embodiment of the present application does not limit this.

[0182] Furthermore, the analysis device can also send a first feature vector set to the management device. When presenting each time series, the management device can present the corresponding feature vector for reference by professionals, thereby assisting professionals in labeling and improving the accuracy of labeling.

[0183] In a second optional method, before sending the time series corresponding to the first feature vector set to the management device in step 308, the feature vectors in the first feature vector set can be clustered to obtain the category relationship of the feature vectors in the first feature vector set; then in step 308, the category relationship and the time series corresponding to the first feature vector set are sent to the management device, so that the management device can present the time series corresponding to the first feature vector set according to the category relationship.

[0184] There are many ways to perform clustering. In one optional implementation, Figure 7 As shown, the process of clustering the feature vectors in the first feature vector set includes:

[0185] Step 3081: Based on the distance between every two eigenvectors in the first eigenvector set, count the neighbor vectors of each eigenvector. The neighbor vectors of any eigenvector in the first eigenvector set are other eigenvectors in the first eigenvector set whose distance to any eigenvector is less than a distance threshold. The distance threshold is a distance specified in multiple distances determined based on the first eigenvector set.

[0186] For example, step 3081 may include the following steps:

[0187] Step B1: The analysis device obtains the distance between every two eigenvectors in the first eigenvector set.

[0188] Assuming that the second reference feature vector and the third reference feature vector are any two reference feature vectors in the reference feature vector set, the second feature is the same feature in the feature corresponding to the second reference feature vector and the feature corresponding to the third reference feature vector, that is, the second feature is the intersection of the feature corresponding to the second reference feature vector and the feature corresponding to the third reference feature vector, the third sub-feature vector is a vector composed of data corresponding to the second feature in the second reference feature vector, and the fourth sub-feature vector is a vector composed of data corresponding to the second feature in the third reference feature vector, then the similarity between the second reference feature vector and the third reference feature vector is the distance between the third sub-feature vector and the fourth sub-feature vector. Wherein, if the categories and number of features involved in the second reference feature vector and the third reference feature vector are different, refer to the first case of the aforementioned step 302, and the method for obtaining the distance between the third sub-feature vector and the fourth sub-feature vector can refer to the aforementioned steps 3021 to 3024; if the categories and number of features involved in the second reference feature vector and the third reference feature vector are the same, refer to the second case of the aforementioned step 302 and directly obtain the distance between the second reference feature vector and the third reference feature vector.

[0189] Step B2: The analyzing device determines a distance threshold among the multiple distances determined based on the first feature vector set.

[0190] Optionally, the analysis device sorts the acquired distances, for example, in ascending or descending order. The distance threshold can be a distance in a specified quantile or a specified order among the sorted distances. The specified quantile or specified order is an empirical value. For example, if the specified quantile is in the top 50% or top 90%, the distance threshold is the distance in the top 50% or top 90% among the sorted distances, where "top" refers to the order from front to back in the arrangement sequence; for example, if the specified order is the 5th, the distance threshold is the distance in the 5th among the sorted distances. For example, assuming the first eigenvector set Z = (z1, z2, z3, z4), the distances between eigenvector z1 and z2, z3, and z4 are 10, 9, and 8, respectively, the distances between eigenvector z2 and z3 and z4 are 11 and 6, respectively, and the distance between z3 and z4 is 5. The quantile is in the top 50%. The analysis device sorts the acquired distances in descending order to obtain the distance sequence: 11, 10, 9, 8, 6, 5. The distance threshold is 9.

[0191] Step B3: Based on the distance between every two eigenvectors in the first eigenvector set, count the neighbor vectors of each eigenvector. The neighbor vectors of any eigenvector in the first eigenvector set are other eigenvectors in the first eigenvector set whose distance to the eigenvector is less than a distance threshold.

[0192] Still taking the example in the aforementioned step B2, the neighbor vector of eigenvector z1 is z4, and the number of neighbor vectors of eigenvector z1 is 1; the neighbor vector of eigenvector z2 is z4, and the number of neighbor vectors is 1; the neighbor vector of eigenvector z3 is z4, and the number of neighbor vectors of eigenvector z3 is 1; the neighbor vectors of eigenvector z4 are z1, z2 and z3, and the number of neighbor vectors of eigenvector z4 is 3.

[0193] Step 3082: Based on the statistical results, classify every two feature vectors whose number of identical neighbor vectors is greater than a threshold as feature vectors of the same type. The threshold is a number specified in the number of neighbor vectors of each feature vector in the first feature vector set.

[0194] For example, step 3082 may include the following steps:

[0195] Step C1: The analysis device obtains the number of neighboring vectors of each feature vector in the first feature vector set.

[0196] Still taking the example in the aforementioned step B2 as an example, the numbers of neighboring vectors of the feature vectors z1, z2, z3, and z4 are 1, 1, 1, and 3 respectively.

[0197] Step C2: The analysis device determines a quantity threshold from the quantity of neighboring vectors of each feature vector in the first feature vector set.

[0198] Optionally, the analysis device sorts the acquired quantities, for example, in ascending or descending order. The quantity threshold may be a quantity in a specified quantile or a specified order among the sorted quantities. The specified quantile or specified order may be an empirical value, for example, the specified quantile may be the top 50% or the top 60%.

[0199] For example, assuming the specified quantile is the top 50%, the analysis device arranges the obtained quantities in descending order to obtain a quantity sequence of 3, 1, 1, 1. The quantity threshold is 1.

[0200] Step C3: Based on the statistical results, every two feature vectors whose number of identical neighbor vectors is greater than a threshold value are classified as feature vectors of the same type.

[0201] Assuming that the quantity threshold is 1, the number of identical neighbor vectors between feature vectors z1, z2, z3, and z4 is 0. Therefore, feature vectors z1, z2, z3, and z4 are respectively classified into one category.

[0202] Assuming that the quantity threshold is 1, the nearest neighbor vectors of the same eigenvectors z1 and z4 are z2 and z3, the nearest neighbor vectors of the same eigenvectors z2 and z3 are z1 and z4, the nearest neighbor vectors of the same eigenvectors z1, z2 and z3 are all empty, and the nearest neighbor vectors of the same eigenvectors z4, z2 and z3 are all empty. Then the eigenvectors z1 and z4 are classified as the same type of eigenvectors, and the eigenvectors z2 and z3 are classified as the same type of eigenvectors.

[0203] In the embodiment of the present application, since the aforementioned distance threshold and quantity threshold are dynamically determined, wherein the distance threshold is a distance specified in multiple distances determined based on the first feature vector set, which reflects the distribution relationship of multiple distances corresponding to the first feature vector set, and is a value that changes with the change of the first feature vector set; the quantity threshold is a number specified in the number of neighboring vectors of each feature vector in the first feature vector set, which reflects the distribution relationship of the number of neighboring vectors of each feature vector in the first feature vector set, and is a value that changes with the change of the first feature vector set. Therefore, the distance threshold and the quantity threshold are relatively changing values, and the category relationship finally obtained by the division based on at least one of the two thresholds is more accurate, can better reflect the correlation between each feature vector, and improve the adaptability of the clustering algorithm.

[0204] It is worth noting that in this second optional method, the analysis device can also send the category relationship to the management device, and the management device can present the time series corresponding to the first feature vector set according to the category relationship. For example, the management device can display multiple time series belonging to the same category on the same user page, and multiple time series belonging to different categories on different user pages; for another example, the management device can display multiple time series belonging to different categories at different locations on the same user page; for another example, the management device displays each time series in correspondence with the category to which it belongs. The management device can present the time series corresponding to the first feature vector set according to the category relationship, which can be used by professionals to refer to the category relationship when labeling, and play a role in assisting professionals in labeling. Based on this, professionals can label time series belonging to the same category with the same label, thereby improving labeling efficiency and increasing labeling accuracy.

[0205] In another optional implementation, a shared nearest neighbor (SNN) algorithm may be used for clustering. Compared to the clustering process provided by the aforementioned optional implementation, the clustering threshold and quantity threshold for clustering using the SNN algorithm are pre-set.

[0206] In another optional implementation, other clustering algorithms may be used for clustering. For example, a clustering algorithm based on a neural network model may be used for clustering. The present embodiment of the application does not limit the algorithm used for clustering.

[0207] For example, Table 2 assumes that clustering is performed on the feature vector of sample data with the ID KPI_2. The time series corresponding to this feature vector is (z1, z2, ..., zn), and it includes data for four features: Moving_average, Weighted_mv, Tsd_seasonal, and period yoy. The time series corresponding to the reference feature vector is (z1, z2, ..., zn), and its class identifier is "1."

[0208] Table 2

[0209]

[0210] Step 309: Receive the time series label corresponding to the first feature vector set sent by the management device.

[0211] After professionals label the time series, the management device receives the labeled labels and sends the labels to the analysis device. The analysis device receives the labels and saves them in correspondence with the corresponding time series.

[0212] In summary, the label determination method provided in the embodiment of the present application migrates labels based on the similarity of the characteristic vectors of the time series, which can realize automatic labeling of sample data and reduce the cost of label determination. And because the similarity calculation is related to the characteristic vector of the time series, the influence of interference information of the time series itself is avoided, for example, the influence of interference information such as sampling period, amplitude change, quadrant drift and noise can be reduced. The accuracy of label determination is improved. In particular, label migration can still be performed accurately in high-dimensional time series. Applying the label determination method provided in the embodiment of the present application to scenarios such as supervised learning algorithms or semi-supervised learning algorithms that require a large amount of labeled sample data can effectively reduce the labeling cost and improve the modeling efficiency of the machine learning model.

[0213] Although traditional label determination methods perform label migration based on the waveform similarity of time series, they cannot perform label migration when the waveforms of some time series themselves are not similar.

[0214] The label determination method provided in the embodiment of the present application uses the similarity of feature vectors for label migration. It is not limited to the label migration of time series with similar waveforms. Label migration can be performed as long as similarity in certain feature dimensions is ensured. Therefore, it can be seen that the embodiment of the present application can be applied to the label migration of time series with different waveforms. Therefore, it can expand the scenarios of label generalization, improve the flexibility and utilization of label migration, and reduce the modeling cost of machine learning models. In particular, in the anomaly detection scenario, label migration between KPIs with certain similar features can be achieved.

[0215] The order of steps in the method for implementing model training provided in the embodiment of the present application can be appropriately adjusted, and the steps can be increased or decreased according to the situation. For example, the aforementioned step 306 can be executed in parallel with other steps. For another example, the analysis device has an input and output interface (such as a user interface), which presents the first time series through the input and output interface and receives the label of the first time series, without executing the interaction process with the management device in steps 304 and 305; or, the analysis device presents the time series corresponding to the first feature vector and the time series corresponding to the second feature vector through the input and output interface, and receives the same label of the time series corresponding to the first feature vector and the time series corresponding to the second feature vector, without executing the interaction process with the management device in steps 3065 and 3066; or, the analysis device presents the time series corresponding to the third feature vector and the time series corresponding to the fourth feature vector through the input and output interface, and receives the same label of the time series corresponding to the third feature vector and the time series corresponding to the fourth feature vector, without executing the interaction process with the management device in steps A5 and A6; or, the analysis device presents the time series corresponding to the first feature vector set through the input and output interface, and receives the label of the time series corresponding to the first feature vector set, without executing the interaction process with the management device in the above steps 308 and 309. Any technical personnel familiar with this technical field can easily think of different methods within the technical scope disclosed in this application, and all of them should be included in the protection scope of this application, so they will not be described in detail.

[0216] The embodiment of the present application provides a label determination device 80, such as Figure 8 As shown, the device includes:

[0217] A first acquisition module 801 is used to acquire a target feature vector of a first time series, where a time series is a set of data arranged in time sequence;

[0218] A second acquisition module 802 is configured to acquire a similarity between the target feature vector and a reference feature vector in a reference feature vector set, wherein the reference feature vector is a feature vector of a second time series with a determined label;

[0219] The determination module 803 is used to determine the label corresponding to the first reference feature vector as the label of the first time series when the similarity between the target feature vector and the first reference feature vector is greater than a similarity threshold, and the first reference feature vector is a reference feature vector in the reference feature vector set.

[0220] In summary, the label determination device provided by the embodiment of the present application, the second acquisition module performs label migration based on the similarity of the characteristic vector of the time series, which can realize automatic labeling of sample data and reduce the cost of label determination. And because the similarity calculation is related to the characteristic vector of the time series, the influence of interference information of the time series itself is avoided, for example, the influence of interference information such as sampling period, amplitude change, quadrant drift and noise can be reduced. The accuracy of label determination is improved. In particular, label migration can still be performed accurately in high-dimensional time series. Applying the label determination device provided by the embodiment of the present application to scenarios such as supervised learning algorithms or semi-supervised learning algorithms that require a large amount of labeled sample data can effectively reduce the labeling cost and improve the modeling efficiency of the machine learning model.

[0221] Optionally, the first time series is a time series of a network key performance indicator KPI.

[0222] Optionally, the reference feature vector includes data of one or more features, and the target feature vector includes data of one or more features.

[0223] The similarity between the target feature vector and the first reference feature vector is the similarity between the first sub-feature vector and the second sub-feature vector, and the first sub-feature vector and the second sub-feature vector are respectively composed of data corresponding to the same features in the target feature vector and the first reference feature vector.

[0224] Optionally, the first sub-feature vector and the second sub-feature vector are both represented in sequence form, data at the same position in the first sub-feature vector and the second sub-feature vector correspond to features of the same category, and the similarity between the first sub-feature vector and the second sub-feature vector is negatively correlated with the distance between the first sub-feature vector and the second sub-feature vector.

[0225] In an alternative approach, Figure 9 As shown, the device 80 further includes:

[0226] a first sending module 804 configured to send the first time series to a management device when the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, so that the management device presents the first time series;

[0227] The first receiving module 805 is configured to receive the label of the first time series sent by the management device.

[0228] In another alternative, such as Figure 10 As shown, the device 80 further includes:

[0229] A third acquisition module 806 is configured to acquire a first feature vector set, wherein the similarity between any feature vector in the first feature vector set and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, and the label of the time series corresponding to any feature vector is undetermined;

[0230] A second sending module 807 is configured to send the time series corresponding to the first feature vector set to a management device, so that the management device can present the time series corresponding to the first feature vector set;

[0231] The second receiving module 808 is configured to receive the labels of the time series corresponding to the first feature vector set sent by the management device.

[0232] Alternatively, as Figure 11 As shown, in Figure 10 On the basis of the above, the device 80 further includes:

[0233] A clustering module 809 is configured to perform clustering processing on the feature vectors in the first feature vector set to obtain a category relationship of the feature vectors in the first feature vector set before sending the time series corresponding to the first feature vector set to the management device;

[0234] The second sending module 807 is configured to:

[0235] The category relationship and the time series corresponding to the first feature vector set are sent to the management device, so that the management device presents the time series corresponding to the first feature vector set according to the category relationship.

[0236] Optionally, the clustering module 809 is configured to:

[0237] Based on the distance between every two feature vectors in the first feature vector set, counting the neighbor vectors of each feature vector, where the neighbor vectors of any feature vector in the first feature vector set are other feature vectors in the first feature vector set whose distance from the any feature vector is less than a distance threshold, where the distance threshold is a distance specified among multiple distances determined based on the first feature vector set;

[0238] Based on the statistical results, every two feature vectors whose number of identical neighbor vectors is greater than a quantity threshold are classified as feature vectors of the same type, where the quantity threshold is a quantity specified in the number of neighbor vectors of each feature vector in the first feature vector set.

[0239] Alternatively, as Figure 12 As shown, the device 80 further includes:

[0240] A fourth acquisition module 810 is configured to acquire a first feature vector of a third time series with a determined label;

[0241] A fifth acquiring module 811 is configured to acquire a similarity between the first feature vector and a reference feature vector in the reference feature vector set;

[0242] The first adding module 812 is configured to add the first feature vector as a reference feature vector to the reference feature set when the similarity between the first feature vector and each reference feature vector in the reference feature vector set is not greater than a storage similarity threshold.

[0243] In an optional implementation, Figure 13 As shown, in Figure 12 On the basis of the above, the device 80 further includes:

[0244] The second adding module 813 is used to add the first feature vector as a reference feature vector to the reference feature set when the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is the same as the label corresponding to the second feature vector.

[0245] In another optional implementation, such as Figure 14 As shown, in Figure 12 On the basis of the above, the device 80 further includes:

[0246] a third sending module 814 configured to, when the similarity between the first feature vector and a second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is different from the label corresponding to the second feature vector, send the time series corresponding to the first feature vector and the time series corresponding to the second feature vector to a management device, so that the management device can present the time series corresponding to the first feature vector and the time series corresponding to the second feature vector;

[0247] A third receiving module 815 is configured to receive the same label of the time series corresponding to the first feature vector and the time series corresponding to the second feature vector sent by the management device;

[0248] An updating module 816 is configured to update, based on the received label, the pre-stored label of the time series corresponding to the first feature vector and the pre-stored label of the time series corresponding to the second feature vector;

[0249] The third adding module 817 is configured to add the first feature vector as a reference feature vector to the reference feature set.

[0250] Optionally, both the target feature vector and the reference feature vector include data of one or more features among statistical features, fitting features or frequency domain features.

[0251] Optionally, the device is applied to a network analyzer, and the label corresponding to the reference feature vector is an anomaly detection label.

[0252] In summary, the label determination device provided by the embodiment of the present application, the second acquisition module performs label migration based on the similarity of the characteristic vector of the time series, which can realize automatic labeling of sample data and reduce the cost of label determination. And because the similarity calculation is related to the characteristic vector of the time series, the influence of interference information of the time series itself is avoided, for example, the influence of interference information such as sampling period, amplitude change, quadrant drift and noise can be reduced. The accuracy of label determination is improved. In particular, label migration can still be performed accurately in high-dimensional time series. Applying the label determination device provided by the embodiment of the present application to scenarios such as supervised learning algorithms or semi-supervised learning algorithms that require a large amount of labeled sample data can effectively reduce the labeling cost and improve the modeling efficiency of the machine learning model.

[0253] Figure 15 This is a block diagram of a label determination device provided in an embodiment of the present application. The label determination device may be an analysis device. Figure 15 As shown, the analysis device 150 includes a processor 1501 and a memory 1502 .

[0254] Memory 1501, used to store computer programs, where the computer programs include program instructions;

[0255] Processor 1502 is used to call a computer program to implement the label determination method provided in the embodiment of the present application.

[0256] Optionally, the network device 150 further includes a communication bus 1503 and a communication interface 1504 .

[0257] The processor 1501 includes one or more processing cores, and the processor 1501 executes various functional applications and data processing by running computer programs.

[0258] The memory 1502 can be used to store computer programs. Optionally, the memory can store an operating system and at least one application unit required for a function. The operating system can be a real-time operating system (RTX), Linux, UNIX, Windows, or OS X.

[0259] There may be multiple communication interfaces 1504, and the communication interfaces 1504 are used to communicate with other storage devices or network devices. For example, in the embodiment of the present application, the communication interface 1504 may be used to receive sample data sent by a network device in the communication network.

[0260] The memory 1502 and the communication interface 1504 are connected to the processor 1501 via the communication bus 1503 respectively.

[0261] An embodiment of the present application provides a computer storage medium having instructions stored thereon. When the instructions are executed by a processor, the label determination method provided in the embodiment of the present application is implemented.

[0262] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0263] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium, or a semiconductor medium (e.g., a solid-state hard disk).

[0264] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A label determination method, characterized in that: The method comprises: Get the target feature vector of the first time series; Obtaining a similarity between the target feature vector and a reference feature vector in a reference feature vector set, where the reference feature vector is a feature vector of a second time series with a determined label; wherein the first time series and the second time series are both time series of a network key performance indicator (KPI), and the target feature vector and the reference feature vector both include data of one or more features selected from the group consisting of a data arrangement period, a data change trend, data fluctuations, statistical features, fitting features, or frequency domain features; When the similarity between the target feature vector and the first reference feature vector is greater than a similarity threshold, determining the label corresponding to the first reference feature vector as the label of the first time series, the first reference feature vector is a reference feature vector in the reference feature vector set, the label of the first time series is used to train a machine learning model, the label is an anomaly detection label, and the trained machine learning model is used to perform anomaly detection on the KPI; The method further comprises: Acquire a first feature vector set, wherein the similarity between any feature vector in the first feature vector set and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, and the label of the time series corresponding to any feature vector is undetermined; Obtaining a distance between every two feature vectors in the first feature vector set, and determining a distance threshold among a plurality of distances determined based on the first feature vector set, wherein the distance threshold is a distance in a specified quantile or a specified order among the sorted distances; Based on the distance between every two feature vectors in the first feature vector set, counting the neighbor vectors of each feature vector, where the neighbor vectors of any feature vector in the first feature vector set are other feature vectors in the first feature vector set whose distance from the any feature vector is less than the distance threshold; Based on the statistical results, every two feature vectors whose number of identical neighbor vectors is greater than a quantity threshold are classified as feature vectors of the same category, thereby obtaining a category relationship of the feature vectors in the first feature vector set; Sending the category relationship and the time series corresponding to the first feature vector set to a management device, so that the management device presents the time series corresponding to the first feature vector set according to the category relationship; Receive a time series label corresponding to the first feature vector set sent by the management device.

2. The method according to claim 1, characterized in that The reference feature vector includes data of one or more features, and the target feature vector includes data of one or more features. The similarity between the target feature vector and the first reference feature vector is the similarity between the first sub-feature vector and the second sub-feature vector, and the first sub-feature vector and the second sub-feature vector are respectively composed of data corresponding to the same features in the target feature vector and the first reference feature vector.

3. The method according to claim 2, characterized in that The similarity between the first sub-feature vector and the second sub-feature vector is negatively correlated with the distance between the first sub-feature vector and the second sub-feature vector.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, sending the first time series to a management device so that the management device presents the first time series; Receive the label of the first time series sent by the management device.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Obtaining a first feature vector of a third time series with a determined label; Obtaining similarity between the first feature vector and a reference feature vector in the reference feature vector set; When the similarity between the first feature vector and each reference feature vector in the reference feature vector set is not greater than a storage similarity threshold, the first feature vector is added to the reference feature set as a reference feature vector.

6. The method according to claim 5, characterized in that The method further comprises: When the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is the same as the label corresponding to the second feature vector, the first feature vector is added to the reference feature set as a reference feature vector.

7. The method according to claim 5, characterized in that The method further comprises: When the similarity between the first feature vector and a second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is different from the label corresponding to the second feature vector, sending the time series corresponding to the first feature vector and the time series corresponding to the second feature vector to the management device, so that the management device presents the time series corresponding to the first feature vector and the time series corresponding to the second feature vector; receiving, from the management device, the same label of the time series corresponding to the first feature vector and the time series corresponding to the second feature vector; Based on the received label, updating the pre-stored label of the time series corresponding to the first feature vector and the label of the time series corresponding to the second feature vector; The first feature vector is added as a reference feature vector to the reference feature set.

8. The method according to any one of claims 1 to 3, characterized in that: The method is applied to a network analyzer.

9. A label determination device, characterized in that: The device comprises: A first acquisition module is used to obtain a target feature vector of a first time series; a second acquisition module, configured to obtain a similarity between the target feature vector and a reference feature vector in a reference feature vector set, wherein the reference feature vector is a feature vector of a second time series for which a label has been determined; wherein the first time series and the second time series are both time series of a network key performance indicator (KPI), and the target feature vector and the reference feature vector both include data of one or more features selected from the group consisting of a data arrangement period, a data change trend, data fluctuations, statistical features, fitting features, or frequency domain features; a determination module, configured to, when a similarity between the target feature vector and a first reference feature vector is greater than a similarity threshold, determine a label corresponding to the first reference feature vector as a label of the first time series, wherein the first reference feature vector is a reference feature vector in the reference feature vector set, the label of the first time series is used to train a machine learning model, the label is an anomaly detection label, and the trained machine learning model is used to perform anomaly detection on the KPI; The device further comprises: a third acquisition module, configured to acquire a first feature vector set, wherein the similarity between any feature vector in the first feature vector set and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, and the label of the time series corresponding to any feature vector is undetermined; a clustering module, configured to obtain the distance between every two feature vectors in the first feature vector set, determine a distance threshold among multiple distances determined based on the first feature vector set, wherein the distance threshold is a distance located at a specified quantile or a specified order in the sorted distances; based on the distance between every two feature vectors in the first feature vector set, count the neighbor vectors of each feature vector, wherein the neighbor vectors of any feature vector in the first feature vector set are other feature vectors in the first feature vector set whose distance from the any feature vector is less than the distance threshold; based on the statistical results, classify every two feature vectors whose number of identical neighbor vectors is greater than the number threshold into the same category of feature vectors, thereby obtaining a category relationship of the feature vectors in the first feature vector set; a second sending module, configured to send the category relationship and the time series corresponding to the first feature vector set to a management device, so that the management device presents the time series corresponding to the first feature vector set according to the category relationship; The second receiving module is configured to receive a label of a time series corresponding to the first feature vector set sent by the management device.

10. The device according to claim 9, characterized in that The reference feature vector includes data of one or more features, and the target feature vector includes data of one or more features. The similarity between the target feature vector and the first reference feature vector is the similarity between the first sub-feature vector and the second sub-feature vector, and the first sub-feature vector and the second sub-feature vector are respectively composed of data corresponding to the same features in the target feature vector and the first reference feature vector.

11. The device according to claim 10, characterized in that The similarity between the first sub-feature vector and the second sub-feature vector is negatively correlated with the distance between the first sub-feature vector and the second sub-feature vector.

12. The device according to any one of claims 9 to 11, characterized in that The device further comprises: a first sending module, configured to, when the similarity between the target feature vector and each reference feature vector in the reference feature vector set is not greater than the similarity threshold, send the first time series to a management device, so that the management device presents the first time series; The first receiving module is configured to receive the label of the first time series sent by the management device.

13. The device according to any one of claims 9 to 11, characterized in that The device further comprises: a fourth acquisition module, configured to acquire a first feature vector of a third time series of which the label has been determined; a fifth acquiring module, configured to acquire a similarity between the first feature vector and a reference feature vector in the reference feature vector set; The first adding module is configured to add the first feature vector as a reference feature vector to the reference feature set when the similarity between the first feature vector and each reference feature vector in the reference feature vector set is not greater than a storage similarity threshold.

14. The device according to claim 13, characterized in that The device further comprises: A second adding module is used to add the first feature vector as a reference feature vector to the reference feature set when the similarity between the first feature vector and the second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is the same as the label corresponding to the second feature vector.

15. The device according to claim 13, characterized in that The device further comprises: a third sending module, configured to, when the similarity between the first feature vector and a second feature vector in the reference feature vector set is greater than the storage similarity threshold, and the label corresponding to the first feature vector is different from the label corresponding to the second feature vector, send the time series corresponding to the first feature vector and the time series corresponding to the second feature vector to a management device, so that the management device can present the time series corresponding to the first feature vector and the time series corresponding to the second feature vector; a third receiving module, configured to receive the same label of the time series corresponding to the first feature vector and the time series corresponding to the second feature vector sent by the management device; An updating module, configured to update, based on the received label, the pre-stored label of the time series corresponding to the first feature vector and the pre-stored label of the time series corresponding to the second feature vector; The third adding module is configured to add the first feature vector as a reference feature vector to the reference feature set.

16. The device according to any one of claims 9 to 11, characterized in that The device is applied to a network analyzer.

17. A label determination device, characterized in that: include: processor and memory; The memory is used to store a computer program, wherein the computer program includes program instructions; The processor is configured to call the computer program to implement the tag determination method according to any one of claims 1 to 8.

18. A computer storage medium, characterized in that The computer storage medium stores instructions, and when the instructions are executed by the processor, the label determination method according to any one of claims 1 to 8 is implemented.

Images